RAG vs fine-tuning: which one do you need?
RAG vs fine-tuning: RAG gives a model facts at query time, fine-tuning changes its behavior. What the research says, what each costs, and when to use which.

RAG and fine-tuning solve different problems. RAG (retrieval-augmented generation) gives a model knowledge at query time by retrieving relevant documents and putting them in the prompt, so facts stay current and citable; fine-tuning changes the model’s weights through extra training, which is good for teaching a format, a style, a task or a vocabulary, and poor at adding new facts. If your problem is “the model doesn’t know our data”, start with RAG; if it is “the model doesn’t behave the way we need”, consider fine-tuning.
What is the difference between RAG and fine-tuning?
The cleanest framing comes from the paper that named RAG. Lewis et al. (22 May 2020) describe two kinds of memory: parametric memory, the knowledge stored in a model’s weights, and non-parametric memory, in their case “a dense vector index of Wikipedia” that the model reads through a retriever. They point out that with parametric memory alone, “providing provenance for their decisions and updating their world knowledge remain open research problems”.
Fine-tuning writes to parametric memory. RAG reads from non-parametric memory. Everything else follows from that:
| RAG | Fine-tuning | |
|---|---|---|
| What changes | The prompt, per request | The model’s weights, once per training run |
| Best at | Facts, documents, data that changes | Format, tone, task skill, domain vocabulary |
| Updating knowledge | Re-index a document | Retrain |
| Citations | Natural: you know which chunk was used | Not available: the fact is somewhere in the weights |
| Access control | Filter what each user can retrieve | Everything trained in is available to every user |
| Running cost | Extra input tokens on every call | Training runs, and hosting if the model is yours |
I explain the RAG side in detail in what is RAG, and the training side in what is fine-tuning.
Can fine-tuning teach a model new facts?
Badly, according to the two papers I’d read first.
Ovadia et al. (10 December 2023) compared unsupervised fine-tuning with RAG for injecting knowledge into LLMs. Fine-tuning helped somewhat, but RAG did better both for knowledge the model had seen in pretraining and for entirely new knowledge. The models struggled to learn new facts through fine-tuning, and the authors suggest that “exposing them to numerous variations of the same fact during training could alleviate this problem”. In other words, one document is not enough; the model needs the fact many ways.
Gekhman et al. (9 May 2024) went further and asked whether fine-tuning on new knowledge causes harm. In a controlled closed-book QA setup, models learned examples with new facts much more slowly than examples consistent with what they already knew, and once they did learn them, those examples “linearly increase the model’s tendency to hallucinate”. Their conclusion is that models get most factual knowledge from pretraining, and fine-tuning mostly teaches them to use it.
That matches the mental model I work with: fine-tuning shapes how a model behaves, retrieval decides what it knows at answer time.
What is fine-tuning actually good for?
Behavior that is hard to describe in a prompt but easy to show with examples:
- A strict output format used millions of times, where a shorter prompt saves real money. Though for JSON, structured outputs usually solve it without training.
- A narrow task like classifying tickets into your 40 categories, where a small fine-tuned model can replace a large prompted one.
- Domain style and vocabulary: clinical note formats, legal drafting conventions, an internal DSL.
- Distilling a large model into a small one for cost or latency.
Parameter-efficient methods made this cheaper. LoRA (Hu et al., 17 June 2021) trains small low-rank matrices instead of the full weights; against full fine-tuning of GPT-3 175B with Adam, the paper reports it can “reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times”. The details are in what is LoRA.
What does each one cost?
RAG’s cost is recurring: retrieved chunks are input tokens on every single call. Fine-tuning’s cost is lumpy: a training run per refresh, plus evaluation, plus hosting if you serve the weights yourself. This script prices RAG’s side with a current list price and shows the other dimension that usually decides it, which is how stale the knowledge gets.
# rag_vs_ft.py: what RAG costs per month, and how stale each approach gets. Python 3, stdlib only.
# Input price: Claude Sonnet 5.5, USD 2.00 per 1M input tokens (Anthropic pricing, read 2026-10-31).
PRICE_IN = 2.00
CHUNK_TOKENS = 5 * 400 # five retrieved chunks added to every prompt
print("RAG's running cost: the retrieved context, billed as input on every call")
for monthly_queries in (10_000, 100_000, 1_000_000):
usd = monthly_queries * CHUNK_TOKENS * PRICE_IN / 1e6
print(f" {monthly_queries:>9,} queries/month -> ${usd:>8,.2f} extra input per month")
print("\nAverage age of the knowledge the model answers from (facts change daily)")
for name, refresh_days in [("RAG, index rebuilt nightly", 1),
("RAG, index updated on write", 0),
("fine-tune, retrained monthly", 30),
("fine-tune, retrained quarterly", 90)]:
print(f" {name:<30} {refresh_days / 2:>5.1f} days")
Output (Python 3.13.11):
RAG's running cost: the retrieved context, billed as input on every call
10,000 queries/month -> $ 40.00 extra input per month
100,000 queries/month -> $ 400.00 extra input per month
1,000,000 queries/month -> $4,000.00 extra input per month
Average age of the knowledge the model answers from (facts change daily)
RAG, index rebuilt nightly 0.5 days
RAG, index updated on write 0.0 days
fine-tune, retrained monthly 15.0 days
fine-tune, retrained quarterly 45.0 days
The chunk size and query volumes are my assumptions; the price is Anthropic’s list price for Claude Sonnet 5.5 as of today. Two readings. At ten thousand queries a month, RAG’s context overhead is a rounding error next to the engineering time a fine-tuning pipeline needs. At a million queries a month, $4,000 of context per month is a real line item, and that is the scale where fine-tuning a smaller model to need less context starts to be worth a spreadsheet. Prompt caching can cut the shared, static part of the prompt, though retrieved chunks usually differ per query.
The staleness numbers are simple arithmetic, which is the point: with periodic retraining, your average answer is half a retraining cycle out of date, and you cannot cite where it came from.
When should I combine RAG and fine-tuning?
When the model needs both your facts and a skill it doesn’t have. RAFT (Zhang et al., 15 March 2024) is the clearest example. It fine-tunes a model for domain-specific RAG: given a question and retrieved documents, the model is trained to ignore the distractor documents that do not help, cite the relevant passage verbatim, and reason in a chain-of-thought style. The authors report consistent gains on PubMed, HotpotQA and Gorilla.
That’s the pattern I’d reach for if a strong RAG setup still produced answers that misuse good context: fine-tune for reading, keep retrieval for knowing.
What I would do
From a data-platform seat, the line I would circle is access control. Coming from data platforms and privacy compliance, the first question I ask of any design is who is allowed to see which record. RAG answers that per request: filter the index by the user’s permissions before retrieval. A fine-tuned model has no such filter; anything in the training data can come out for anyone, and you cannot delete one person’s record from the weights when a deletion request arrives. For any data covered by privacy rules, that alone settles it in favor of retrieval.
The order I’d follow:
- Prompt a strong model first. Many “we need fine-tuning” problems are prompt problems.
- Add RAG for knowledge. Measure it with the methods in how to evaluate RAG.
- If evals show the model has the right context but uses it badly, or formats inconsistently at a volume where it matters, fine-tune, ideally with LoRA on a smaller model.
- Keep retrieval in front of the fine-tuned model for anything that changes.
And if the corpus is small enough to fit in the prompt, check RAG vs long context before building either.
FAQ
Is RAG better than fine-tuning?
For adding knowledge, yes: Ovadia et al. found RAG outperformed unsupervised fine-tuning for both previously seen and new knowledge. For changing behavior, format or style, fine-tuning is the right tool.
Does fine-tuning reduce hallucinations?
Not when it adds new facts. Gekhman et al. found that once models learned fine-tuning examples with new knowledge, those examples linearly increased hallucination. Fine-tuning on behavior can reduce format errors, which is different.
Can I use RAG and fine-tuning together?
Yes. RAFT fine-tunes a model to read retrieved documents better, ignoring distractors and citing the relevant passage, while retrieval still supplies the facts.
Is RAG cheaper than fine-tuning?
At low and medium volume, almost always, because there is no training run or model hosting. At very high volume, the extra input tokens per call add up, and a fine-tuned smaller model can come out cheaper. Price both on your own traffic.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv, 22 May 2020
- Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, arXiv, 10 December 2023
- Gekhman et al., Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?, arXiv, 9 May 2024
- Zhang et al., RAFT: Adapting Language Model to Domain Specific RAG, arXiv, 15 March 2024
- Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, arXiv, 17 June 2021
- Anthropic, Pricing (Claude Sonnet 5.5 input price read 2026-10-31)