What is RAG? Retrieval-augmented generation, explained
RAG (retrieval-augmented generation) fetches relevant text from your data and puts it in the prompt so the LLM answers from it. How it works, in Python.

RAG (retrieval-augmented generation) is a pattern where an application first searches your own data for passages relevant to a question, then pastes those passages into the prompt so a language model answers from them instead of from memory. The model stays frozen; what changes per request is the context you hand it. That makes answers current, specific to your documents and checkable against a source.
The term comes from a 2020 paper by Lewis et al. at Facebook AI Research, UCL and NYU, published at NeurIPS 2020. Five years later it is the default way enterprises put an LLM on top of their own documents, and most of the work in a RAG system is search engineering, not prompting.
How does RAG work?
Every RAG system has the same three steps, whatever the framework calls them:
- Index. Split your documents into chunks, turn each chunk into something searchable (a vector, a keyword index, or both) and store it. This happens ahead of time.
- Retrieve. When a question arrives, search the index and keep the top few chunks.
- Generate. Put those chunks into the prompt with an instruction like “answer only from these sources, and cite them”, then call the model.
The original paper describes this as combining two kinds of memory. The model’s weights are “parametric memory”: knowledge baked in during training. The search index is “non-parametric memory”: a store you can read, edit and replace without retraining. In their setup the parametric memory was BART-large (a 400M parameter seq2seq model), and the non-parametric memory was a December 2018 Wikipedia dump split into disjoint 100-word chunks, 21M of them, embedded with the DPR retriever and searched with FAISS. They retrieved 5 or 10 documents per query.
One experiment in that paper is the whole argument for RAG in a single result. The authors built a second index from a December 2016 Wikipedia dump and asked the model about 82 world leaders who had changed between 2016 and 2018. With the matching index, it answered 70% (2016) and 68% (2018) correctly. With the mismatched index, accuracy fell to 12% and 4%. Same weights, different documents, different answers. You update what a RAG system knows by swapping its index.
What does RAG look like in code?
Here is the loop with everything but the model call, in Python 3 standard library only. It splits three policy documents into sentence chunks, scores each chunk against the question with a TF-IDF bag-of-words vector (a stand-in for an embedding model, which would need a download), keeps the top two and builds the prompt.
# rag.py: the retrieve-then-generate loop, minus the model call. Python 3 stdlib only.
import math, re
from collections import Counter
# 1. Index: split documents into chunks and store a vector per chunk.
corpus = {
"leave-policy.md": "Employees get 24 days of paid leave per year. Unused leave carries over, up to 10 days. "
"Leave requests go to your manager in the HR portal at least 5 working days ahead.",
"expenses.md": "Submit expense claims within 30 days with a receipt. Claims above 5,000 rupees need "
"director approval. Reimbursement lands with the next salary cycle.",
"it-security.md": "Laptops must use full-disk encryption. Report a lost device to IT within 1 hour. "
"Passwords rotate every 90 days unless you use a passkey.",
}
def words(t): return re.findall(r"[a-z0-9]+", t.lower())
chunks = [] # (source, text)
for src, text in corpus.items():
for sentence in re.split(r"(?<=\.)\s+", text):
chunks.append((src, sentence))
df = Counter(w for _, c in chunks for w in set(words(c)))
def vec(t): # TF-IDF bag of words: a stand-in for an embedding model
tf = Counter(words(t))
return {w: n * math.log(1 + len(chunks) / df.get(w, 0.5)) for w, n in tf.items()}
index = [(src, c, vec(c)) for src, c in chunks]
def cosine(a, b):
dot = sum(a[w] * b.get(w, 0) for w in a)
na, nb = math.sqrt(sum(x * x for x in a.values())), math.sqrt(sum(x * x for x in b.values()))
return dot / (na * nb) if na and nb else 0.0
# 2. Retrieve: score every chunk against the question, keep the top k.
def retrieve(question, k=2):
q = vec(question)
return sorted(((cosine(q, v), src, c) for src, c, v in index), reverse=True)[:k]
# 3. Augment: paste the retrieved chunks into the prompt with their sources.
question = "How many leave days can I carry over to next year?"
hits = retrieve(question)
context = "\n".join(f"[{i}] ({src}) {c}" for i, (_, src, c) in enumerate(hits, 1))
prompt = (f"Answer using only the sources below. Cite them like [1]. "
f"If they do not contain the answer, say so.\n\n{context}\n\nQuestion: {question}")
print(f"{len(chunks)} chunks indexed")
for s, src, c in hits:
print(f" {s:.3f} {src}: {c[:60]}")
print("\n--- prompt sent to the model ---\n" + prompt)
Output (Python 3.14.5):
9 chunks indexed
0.227 leave-policy.md: Unused leave carries over, up to 10 days.
0.163 leave-policy.md: Employees get 24 days of paid leave per year.
--- prompt sent to the model ---
Answer using only the sources below. Cite them like [1]. If they do not contain the answer, say so.
[1] (leave-policy.md) Unused leave carries over, up to 10 days.
[2] (leave-policy.md) Employees get 24 days of paid leave per year.
Question: How many leave days can I carry over to next year?
That prompt is the entire trick. Send it to any chat model and the answer will be “up to 10 days [1]”, because the fact is sitting in the context. The model never saw leave-policy.md in training and doesn’t need to.
Swap the toy parts for real ones and you have a production skeleton: an embedding model instead of vec, a vector database instead of the index list, a real chunker instead of a sentence split, and an LLM call at the end.
Why does RAG matter to a builder?
Three reasons, in the order I’d weigh them.
Freshness and ownership. Model weights freeze at a training cutoff and know nothing about your HR policy. An index can be rebuilt nightly, or updated on every write. The Lewis paper’s index swap is the clean demonstration.
Citations. Because the answer is built from named chunks, you can show the user where each claim came from and let them click through. That is the difference between a demo and something a compliance team will sign off. From the data-platform and privacy seat I sat in at Walmart, the property I’d circle is that access control lives in the retriever: filter chunks by what this user may read before they ever reach the prompt, and the model can’t leak what it never received.
Cost. Fine-tuning a model on your documents is slow to repeat and doesn’t give you citations. Retrieval adds one search call and some input tokens. Fine-tuning is better at changing behavior and format than at adding facts.
Doesn’t long context make RAG unnecessary?
Not by itself. Gemini 1.5 showed million-token windows in 2024, and if your whole knowledge base fits comfortably, putting all of it in the prompt is a valid design. Anthropic’s contextual retrieval post says as much: under about 200,000 tokens (roughly 500 pages), you can skip RAG and include everything, with prompt caching to keep it affordable.
Past that size, every request pays for every token, and latency grows with the prompt. Retrieval is how you send 2,000 relevant tokens instead of 2 million mostly irrelevant ones. The trade-off is cost, latency and freshness, which I work through in RAG vs long context.
Where does RAG break?
Most RAG failures trace back to retrieval, not generation. If the right chunk isn’t in the top k, the model can’t answer, and a well-behaved model will say so while a less careful one will improvise.
Chunking cuts facts in half. Split mid-sentence and the answer lives in two chunks, neither of which matches the question well. See RAG chunking strategies.
Vectors miss exact tokens. Embeddings are good at meaning and weak at identifiers like ERR-4012, SKU codes and names. Adding keyword search with BM25 and fusing the two lists is called hybrid RAG, and it is the first upgrade I’d make.
Chunks lose their document. A chunk that says “revenue grew 3%” doesn’t say which company. Contextual retrieval fixes that by prepending a short, LLM-written description of where each chunk sits.
Top-k is too coarse. A first-stage retriever is built for speed. A reranker reads the question and each candidate together and reorders the shortlist.
Questions don’t look like answers. A short question and a long paragraph sit in different parts of the vector space. HyDE searches with a generated answer instead.
The source text is hostile. Retrieved chunks are untrusted input. A document that says “ignore previous instructions” lands in the prompt with everything else, which is prompt injection by another route.
What I would do
For a first RAG system over internal documents:
- Start with structure-aware chunking (split on headings and paragraphs), a few hundred tokens per chunk, and keep the document title and section path with every chunk.
- Index the same chunks twice: BM25 and an embedding model. Retrieve 20 to 50 from each, fuse with reciprocal rank fusion, then rerank down to 5 to 10.
- Write a prompt that requires citations and permits “I don’t know”.
- Build a set of 50 real questions with known source chunks before tuning anything, and measure recall at k on retrieval separately from answer quality. How to evaluate RAG covers the metrics.
- Apply permissions in the retrieval filter, never in the prompt.
When I built rSearch, retrieval was a web search API (Serper) rather than a vector store, and the generation side was DeepSeek R1. The shape was the same three steps. Whether the index is yours or Google’s, the quality ceiling is set by what comes back from search.
FAQ
What does RAG stand for in AI?
Retrieval-augmented generation. The name comes from Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020).
Does RAG need a vector database?
No. You need a search index. Vectors are common, but BM25 keyword search, SQL, a web search API or a mix all work. Small corpora can live in memory, as in the script above.
Does RAG stop hallucinations?
It reduces them on questions your documents answer, because the facts are in the prompt. It doesn’t remove them: if retrieval returns the wrong chunks, or none, the model can still make things up. Require citations and allow “I don’t know”.
What is the difference between RAG and fine-tuning?
RAG changes what the model reads at request time; fine-tuning changes the model’s weights. Use RAG for facts that change or need citations, and fine-tuning for behavior, tone or format.
What is agentic RAG?
A setup where the model decides when to search, what to search for and whether to search again, using retrieval as a tool in a loop. See agentic RAG.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv May 22, 2020, NeurIPS 2020 (model setup in section 2, Wikipedia index in section 3, index hot-swapping in section 4)
- Anthropic, Introducing Contextual Retrieval, September 19, 2024