What is contextual retrieval? Anthropic's RAG fix, explained
Contextual retrieval prepends an LLM-written note on where each chunk sits in its document before indexing, cutting RAG retrieval failures. With Python.

Contextual retrieval is a RAG indexing technique where, before a chunk is embedded or added to a keyword index, an LLM reads the whole source document and writes a short description of where that chunk fits (which document, which company, which section), and that description is prepended to the chunk. Retrieval then works on chunks that carry their own context. Anthropic introduced it on September 19, 2024 and reported that it cut failed retrievals by 49%, and by 67% with a reranker added.
It fixes a problem every RAG system has and few people name: chunking strips context. If you haven’t read it, RAG chunking strategies shows how that happens.
What problem does contextual retrieval solve?
Anthropic’s own example is the clearest. A chunk from an SEC filing says:
“The company’s revenue grew by 3% over the previous quarter.”
Which company? Which quarter? The document knows; the chunk doesn’t. A question like “What was ACME’s revenue growth in Q2 2023?” has nothing to match on except “revenue” and “grew”, and so does every other filing in the index. With contextual retrieval the indexed text becomes, in their example, something that starts “This chunk is from an SEC filing on ACME corp’s performance in Q2 2023;” followed by the previous quarter’s revenue figure and then the original sentence.
Both halves of a hybrid retriever benefit. The embedding now encodes ACME and Q2 2023, and BM25 can match “ACME” as an exact, rare term. Anthropic calls the two parts Contextual Embeddings and Contextual BM25.
How does contextual retrieval work?
At index time, for every chunk:
- Send the model the full document and the chunk, with an instruction to write a short context that situates the chunk within the document for search, and to answer with only that context. (Anthropic’s exact prompt is in the post; the closing line is “Answer only with the succinct context and nothing else.”)
- Take the reply, which the post says is usually 50 to 100 tokens.
- Prepend it to the chunk text.
- Embed the result and add it to the BM25 index, exactly as you would a normal chunk.
Nothing changes at query time. The retrieval code, the reranker and the prompt to the answering model all stay the same; only the indexed text is richer.
The obvious objection is cost: you are sending the whole document to an LLM once per chunk. Prompt caching is what makes it affordable. The document goes in a cached prefix, so each additional chunk from the same document reuses it. Anthropic used Claude 3 Haiku and estimated “the one-time cost to generate contextualized chunks is $1.02 per million document tokens”, assuming 800-token chunks, 8,000-token documents and 100 tokens of context per chunk. The same post says caching cuts latency by more than 2x and costs by up to 90%. My prompt caching explainer covers the mechanism.
What does it do in code?
This script indexes four chunks from two quarterly reports with BM25, once as-is and once with a context prefix. The prefixes are hand-written, standing in for the LLM call, so the script runs offline. Then it prices contextualizing two corpus sizes at Anthropic’s published rate.
# contextual.py: why a chunk needs its document's context. Python 3 stdlib only.
import math, re
from collections import Counter
# Chunks cut from two quarterly reports. On their own, none says which company it is about.
chunks = {
"acme-q2#3": "Revenue grew by 3% over the previous quarter, driven by enterprise renewals.",
"acme-q2#7": "Operating margin narrowed as hiring outpaced revenue.",
"globex-q2#3": "Revenue grew by 11% over the previous quarter on strong retail demand.",
"globex-q2#5": "The company expects revenue growth to slow in the second half.",
}
# What an LLM writes when shown the whole document plus one chunk (Anthropic's prompt asks for
# "a short succinct context to situate this chunk"). Hand-written here so the demo runs offline.
context = {
"acme-q2#3": "From ACME Corp's Q2 2023 quarterly report, results section.",
"acme-q2#7": "From ACME Corp's Q2 2023 quarterly report, costs section.",
"globex-q2#3": "From Globex's Q2 2023 quarterly report, results section.",
"globex-q2#5": "From Globex's Q2 2023 quarterly report, outlook section.",
}
def words(t): return re.findall(r"[a-z0-9]+", t.lower())
def bm25_rank(texts, query, k1=1.2, b=0.75):
W = {d: words(t) for d, t in texts.items()}
N, avg = len(W), sum(map(len, W.values())) / len(W)
df = Counter(w for ws in W.values() for w in set(ws))
def score(d):
s = 0.0
for w in words(query):
tf = W[d].count(w)
if tf:
s += math.log(1 + (N - df[w] + 0.5) / (df[w] + 0.5)) * tf * (k1 + 1) / (tf + k1 * (1 - b + b * len(W[d]) / avg))
return s
return [(d, round(score(d), 2)) for d in sorted(texts, key=score, reverse=True)]
query = "ACME revenue over the previous quarter, Q2 2023"
print("plain chunks: ", bm25_rank(chunks, query)[:2])
contextual = {d: f"{context[d]} {t}" for d, t in chunks.items()}
print("contextual chunks:", bm25_rank(contextual, query)[:2])
# One-time cost, using Anthropic's published estimate for contextualizing a corpus
# with prompt caching: $1.02 per million document tokens (Sep 2024, Claude 3 Haiku).
for corpus_tokens in (2_000_000, 50_000_000):
print(f"{corpus_tokens:>11,} document tokens -> ${corpus_tokens / 1e6 * 1.02:,.2f} one-time")
Output (Python 3.14.5):
plain chunks: [('acme-q2#3', 2.4), ('globex-q2#3', 2.4)]
contextual chunks: [('acme-q2#3', 3.31), ('globex-q2#3', 2.7)]
2,000,000 document tokens -> $2.04 one-time
50,000,000 document tokens -> $51.00 one-time
On plain chunks, ACME’s and Globex’s “revenue grew over the previous quarter” chunks tie at 2.4. Neither mentions a company, so BM25 has nothing to separate them; the order shown is just dictionary order. With the prefixes, the ACME chunk scores 3.31 against Globex’s 2.7, because “ACME” now appears in exactly one chunk and carries a high IDF. An embedding model sees the same shift in meaning.
The cost lines use the September 2024 Haiku estimate, so treat them as a rough order of magnitude: about a dollar per million tokens, paid once at index time and again only for documents that change. Check current model prices before you budget.
How much does contextual retrieval help?
Anthropic measured “1 minus recall@20”: the share of relevant documents that did not show up in the top 20 retrieved chunks. They tested codebases, fiction, arXiv papers and science papers, with several embedding models (Gemini and Voyage embeddings did best). The headline numbers below are the post’s averages at top 20, starting from a 5.7% failure rate:
| Setup | Failure rate | Reduction |
|---|---|---|
| Baseline, no contextualization | 5.7% | |
| Contextual embeddings | 3.7% | 35% |
| Contextual embeddings + contextual BM25 | 2.9% | 49% |
| Same, plus reranking (top 150 to top 20) | 1.9% | 67% |
The reranker was Cohere’s. Three findings from the post I’d carry into any design: contextual retrieval helped with every embedding model they tested; passing the top 20 chunks to the answering model beat top 10 and top 5; and the gains stack, so contextualization, hybrid search and reranking are complements rather than alternatives.
When should you skip it?
When you don’t need RAG at all. The same post says that if your knowledge base is under about 200,000 tokens (roughly 500 pages), you can put the whole thing in the prompt and use caching, with no retrieval. I compare the two approaches in RAG vs long context.
Also skip or deprioritize it when chunks already carry their context: product records with names in every row, support articles where each chunk is a whole short article, or structure-aware chunks that already prefix the title and heading path. Prepending “Leave policy > Carry over” costs nothing and gets you part of the benefit for free.
Where does contextual retrieval break?
- Indexing cost and time scale with the corpus. One LLM call per chunk is cheap per million tokens and slow at a billion. Re-running it after changing the prompt or the chunker means re-indexing everything.
- Very long documents. The whole document has to fit in the model’s context alongside the chunk. Past that, you need a summary of the document or of the neighboring sections instead.
- The context can be wrong. An LLM summarizing where a chunk fits can misattribute it, especially in documents that discuss several entities. A wrong prefix is worse than none, because it actively pulls the chunk toward the wrong queries. Spot-check a sample.
- Freshness. Every update to a document means regenerating context for its changed chunks, and possibly all of them if the change alters what other chunks refer to.
- Privacy. Every document passes through an LLM at index time. For regulated data, the model, region and retention terms of that call need the same review as the answering model. From the privacy compliance work I did at Walmart, that’s the line item that would hold up a launch, not the dollar cost.
What I would do
- Start with free context: structure-aware chunks with the document title and heading path prefixed.
- Add hybrid retrieval and a reranker. These need no index-time LLM calls.
- Measure recall at 20 on a judged query set. If failures cluster on chunks that don’t name their subject (pronouns, “the company”, “this section”), add contextual prefixes.
- Use a small, cheap model with prompt caching, store the generated context as its own field next to the original chunk text, and version the prompt so you can regenerate selectively.
- Show the original chunk, not the prefixed one, when citing sources to users.
FAQ
What is the difference between contextual retrieval and normal RAG?
Normal RAG indexes chunks as they were cut. Contextual retrieval prepends an LLM-written description of each chunk’s place in its document before indexing. Query time is the same.
Does contextual retrieval work with BM25?
Yes. Anthropic indexes the contextualized chunks in BM25 too (“Contextual BM25”), and combining it with contextual embeddings gave a larger improvement (49%) than contextual embeddings alone (35%).
How much does contextual retrieval cost?
Anthropic’s September 2024 estimate, with Claude 3 Haiku and prompt caching, was $1.02 per million document tokens, paid once at indexing. Current costs depend on the model you use.
Is contextual retrieval the same as late chunking?
No, but they target the same problem. Late chunking embeds the whole document with a long-context embedding model and pools per chunk afterwards; contextual retrieval adds text written by an LLM. Late chunking needs no generation step; contextual prefixes also help keyword search.
Sources
- Anthropic, Introducing Contextual Retrieval, September 19, 2024 (method, prompt, cost estimate, results)
- Günther et al., Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models, arXiv September 7, 2024
- Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond, 2009