RAG vs long context: do 1M-token windows replace RAG?
RAG vs long context: 1M-token windows can hold a whole corpus, but cost, latency and lost-in-the-middle effects still favor retrieval. Research and cost math.

RAG vs long context is a choice between retrieving a few relevant chunks into the prompt and putting the entire corpus into a very large context window. Research shows long-context models often answer slightly better when they get everything, but RAG is far cheaper and faster per question, scales past any window, and handles data that changes, so 1M-token windows shift the trade-off without replacing retrieval. A common answer in 2026 is both: retrieve by default and fall back to long context for the questions retrieval can’t handle.
Why is this even a question?
Because context windows grew faster than anyone expected. Google’s Gemini 1.5 put a million tokens in front of developers in 2024, and by now it is the standard size. Anthropic’s models overview lists a 1M-token window for every current Claude model, Haiku 5.5 included, and says 1M tokens is roughly 555k words on the current tokenizer. The pricing page adds that Claude 4.6 and later models other than Haiku 5.5 bill the full window at standard rates: “A 900k-token request is billed at the same per-token rate as a 9k-token request.”
If a whole policy manual, codebase or contract set fits in one prompt, why build a retrieval pipeline at all? The context window explainer covers what the window is; this post is about whether filling it is a good idea.
Does long context beat RAG on quality?
When cost is ignored, often yes. Li et al. (23 July 2024) benchmarked Gemini-1.5-Pro, GPT-4O and GPT-3.5-Turbo and found that with sufficient resources, “LC consistently outperforms RAG in terms of average performance”. Their RAG baseline retrieved k = 5 chunks of 300 words each.
The more interesting finding is in the details. For 63 percent of queries, the RAG and long-context predictions were exactly identical. So for most questions, paying for the whole corpus bought nothing. That observation led to their method, Self-Route: send every query through RAG first, let the model judge whether the retrieved chunks are enough to answer, and only send the unanswerable ones to long context. It kept performance comparable to long context while cutting cost by 65 percent for Gemini-1.5-Pro and 39 percent for GPT-4O.
Do models actually use everything in a long context?
Not evenly. Lost in the Middle (Liu et al., 6 July 2023, published in TACL) showed that models perform best when the relevant information is near the beginning or end of the input, and noticeably worse when it sits in the middle, even for models built for long contexts.
RULER (Hsieh et al., 9 April 2024) showed the gap between advertised and usable length. It evaluated 17 long-context models on 13 tasks that go beyond the single needle-in-a-haystack test, adding multiple needles, multi-hop tracing and aggregation. All 17 claimed context sizes of 32K tokens or more, but “only half of them can maintain satisfactory performance at the length of 32K”. Near-perfect needle scores did not predict performance on harder tasks.
Models have improved a lot since those papers, and I would not quote their numbers as limits of 2026 models. What has not changed is the method lesson: a model’s advertised window tells you what it accepts, not what it reliably uses. Test your actual task at your actual length.
What does each approach cost per question?
This is where the trade-off usually gets decided. The script prices one question three ways on Claude Sonnet 5.5 at today’s list prices: the full 400k-token corpus in the prompt, the same with the corpus served from the prompt cache, and RAG with five 400-token chunks.
# rag_vs_lc.py: cost per question, whole corpus in context vs retrieved chunks.
# Python 3, stdlib only. Claude Sonnet 5.5 list prices, USD per 1M tokens,
# from Anthropic's pricing page as read on 2026-10-31.
IN, CACHE_WRITE_5M, CACHE_HIT, OUT = 2.00, 2.50, 0.10, 10.00
CORPUS = 400_000 # tokens: the whole document set (fits in a 1M window)
PROMPT = 300 # tokens: instructions plus the question
CHUNKS, CHUNK = 5, 400
ANSWER = 500 # output tokens per answer
QUERIES = 10_000
def usd(tokens, price):
return tokens * price / 1e6
out = usd(ANSWER, OUT)
setups = {
"long context, no cache": usd(CORPUS + PROMPT, IN) + out,
"long context, cache hit": usd(CORPUS, CACHE_HIT) + usd(PROMPT, IN) + out,
"RAG, top-5 chunks": usd(CHUNKS * CHUNK + PROMPT, IN) + out,
}
print(f"{'setup':<24} {'per question':>12} {'per 10k':>10}")
for name, cost in setups.items():
print(f"{name:<24} {cost:>12.4f} {cost * QUERIES:>10,.2f}")
# The cache has to be written first, and a 5-minute cache expires when traffic is quiet.
write = usd(CORPUS, CACHE_WRITE_5M)
print(f"\none cache write of the corpus: ${write:.2f}")
print(f"same corpus as plain input: ${usd(CORPUS, IN):.2f}")
print(f"long context (cached) / RAG cost ratio: "
f"{setups['long context, cache hit'] / setups['RAG, top-5 chunks']:.1f}x")
Output (Python 3.13.11):
setup per question per 10k
long context, no cache 0.8056 8,056.00
long context, cache hit 0.0456 456.00
RAG, top-5 chunks 0.0096 96.00
one cache write of the corpus: $1.00
same corpus as plain input: $0.80
long context (cached) / RAG cost ratio: 4.8x
Without caching, stuffing the corpus costs about 84 times more per question than RAG ($0.8056 against $0.0096), because every question re-reads 400,000 tokens. Prompt caching changes the picture a lot: cache reads on Sonnet 5.5 are $0.10 per million, so the cached version is only 4.8 times the RAG cost. That makes long context realistic for a stable corpus with steady traffic. A cache that expires during a quiet period has to be written again at $1.00, so bursty traffic pays more.
Cost is not the only axis. Every token still has to be processed, and time to first token grows with prompt length; I go through the latency side in LLM latency explained. The corpus size and chunk counts are my assumptions, so rerun the script with your own numbers.
When should I use long context instead of RAG?
Long context wins when:
- The corpus fits and is stable. One contract, one codebase snapshot, one book. Cache it and ask many questions.
- The question needs the whole document. “Summarize every obligation in this agreement” or “find inconsistencies between these sections” has no top-k answer; it is the global-question problem that GraphRAG also attacks.
- Building retrieval is the bottleneck. For a prototype or a one-off analysis, pasting the documents is the fastest route to an answer.
RAG wins when:
- The corpus is bigger than any window, or grows. Most enterprise document stores are.
- Data changes. Re-indexing a document is cheap; rebuilding a cached prompt prefix on every change is not.
- Users have different permissions. Retrieval can filter per user. A shared cached corpus cannot.
- Volume is high. At hundreds of thousands of questions, the per-question gap above becomes the budget.
What I would do
From a data-platform seat, the line I would circle is permissions and freshness, not quality. Most of the enterprise corpora I’d point an LLM at are too large for any window, change daily and carry access rules. That rules out “put everything in the prompt” before quality even comes up.
So I’d build retrieval first, with hybrid search and a reranker, and use long context as a targeted tool: load the full text of the two or three documents retrieval identified, instead of five chunks from them. That gets most of the long-context quality benefit at a fraction of the tokens. Then I’d copy the Self-Route idea: when the model says the retrieved context is insufficient, escalate to a bigger context or to an agentic RAG loop, and log how often that happens. If the escalation rate is high, the retrieval needs work.
For a single stable document set that fits in the window, I’d skip RAG entirely and use caching. Building a retrieval pipeline for a 200-page manual is engineering for its own sake.
FAQ
Does long context make RAG obsolete?
No. Li et al. found long context slightly better on average when cost is ignored, but RAG gave identical answers on 63 percent of queries at far lower cost. Corpora larger than the window, changing data and per-user permissions still need retrieval.
What is the lost in the middle problem?
Liu et al. found that language models use information best when it is at the start or end of a long input, and noticeably worse when it is in the middle. It is a reason to put the most relevant material at the edges of the prompt.
What is Self-Route?
A hybrid from Li et al. (2024): answer with RAG first, have the model judge whether the retrieved context is enough, and send only the unanswerable queries to long context. It cut cost by 65 percent for Gemini-1.5-Pro and 39 percent for GPT-4O while staying close to long-context performance.
Is a bigger context window always better?
A bigger window raises the ceiling on what you can send, not the reliability of what the model does with it. RULER found that half of 17 long-context models fell short at 32K tokens despite claiming more. Test at the length you plan to use.
Sources
- Li et al., Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach, arXiv, 23 July 2024 (v2 17 October 2024)
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts, arXiv, 6 July 2023 (TACL)
- Hsieh et al., RULER: What’s the Real Context Size of Your Long-Context Language Models?, arXiv, 9 April 2024
- Anthropic, Models overview and Pricing (read 2026-10-31)