How to evaluate RAG: retrieval and generation metrics
How to evaluate RAG: measure retrieval with recall@k, MRR and nDCG, and answers with faithfulness and relevance. The metrics explained, with runnable Python.

To evaluate a RAG system, score its two halves separately: retrieval (did the right chunks come back, and how high were they ranked?) with metrics like recall@k, MRR and nDCG against labeled question-to-chunk pairs, and generation (is the answer supported by those chunks and does it address the question?) with faithfulness and answer relevance, usually judged by an LLM. Then score the whole pipeline end to end on a fixed test set, and rerun that set on every change.
Why evaluate retrieval and generation separately?
Because they fail for different reasons and the fixes live in different places. A wrong answer can come from a retriever that never fetched the right passage, or from a model that had the right passage and ignored or embellished it. An end-to-end score alone cannot tell you which, so you cannot tell whether to work on chunking, on search, or on the prompt.
The research frameworks split it the same way. Ragas (Es et al., 26 September 2023) evaluates three dimensions: whether retrieval finds relevant, focused context, whether the LLM uses that context faithfully, and the quality of the generation itself, and it is designed to work “without having to rely on ground truth human annotations”. ARES (Saad-Falcon et al., 16 November 2023) scores context relevance, answer faithfulness and answer relevance, using small fine-tuned judge models trained on synthetic data, plus “a few hundred human annotations” to correct the judges’ errors with prediction-powered inference.
Which retrieval metrics should I use?
Retrieval is the cheap half to measure, because the labels are just IDs. For each test question you record which chunk IDs actually contain the answer. Then for each retriever you compare its ranked list against those labels.
- Recall@k: of the relevant chunks, what share appeared in the top k? This is the metric I watch first, because the model cannot use a chunk it never sees. Pick k equal to what you actually put in the prompt.
- MRR (mean reciprocal rank): one over the rank of the first relevant chunk, averaged across questions. It rewards putting the answer at position 1.
- nDCG@k (normalized discounted cumulative gain): credits every relevant chunk, discounted by log of its position, divided by the best possible score. It is the standard ranking metric when a question has several relevant chunks.
- Precision@k: of the top k, what share was relevant? Useful when irrelevant context distracts the model or costs real money in tokens.
Here they are, computed for two retrievers over four labeled questions, plus a crude groundedness check on one answer.
# rag_eval.py: retrieval metrics plus a crude groundedness check. Python 3, stdlib only.
import math, re
# Gold labels: which chunk IDs actually answer each question.
gold = {"q1": {"c7"}, "q2": {"c2", "c9"}, "q3": {"c4"}, "q4": {"c11"}}
# Ranked chunk IDs from two retrievers you want to compare.
runs = {
"vector only": {"q1": ["c3", "c7", "c1", "c8", "c5"], "q2": ["c2", "c6", "c1", "c3", "c8"],
"q3": ["c5", "c6", "c1", "c2", "c4"], "q4": ["c1", "c2", "c3", "c5", "c6"]},
"hybrid+rerank": {"q1": ["c7", "c3", "c1", "c8", "c5"], "q2": ["c9", "c2", "c6", "c1", "c3"],
"q3": ["c4", "c5", "c6", "c1", "c2"], "q4": ["c3", "c11", "c1", "c2", "c5"]},
}
def recall_at_k(ranked, rel, k):
return len(set(ranked[:k]) & rel) / len(rel)
def mrr(ranked, rel):
return next((1 / (i + 1) for i, d in enumerate(ranked) if d in rel), 0.0)
def ndcg_at_k(ranked, rel, k):
dcg = sum(1 / math.log2(i + 2) for i, d in enumerate(ranked[:k]) if d in rel)
ideal = sum(1 / math.log2(i + 2) for i in range(min(len(rel), k)))
return dcg / ideal
print(f"{'retriever':<14} {'recall@3':>8} {'recall@5':>8} {'MRR':>6} {'nDCG@5':>7}")
for name, run in runs.items():
rows = [(recall_at_k(run[q], gold[q], 3), recall_at_k(run[q], gold[q], 5),
mrr(run[q], gold[q]), ndcg_at_k(run[q], gold[q], 5)) for q in gold]
avg = [sum(col) / len(rows) for col in zip(*rows)]
print(f"{name:<14} {avg[0]:>8.2f} {avg[1]:>8.2f} {avg[2]:>6.2f} {avg[3]:>7.2f}")
# Groundedness: share of answer sentences whose content words all appear in the context.
# A lexical stand-in for an LLM judge that checks each claim against the context.
context = "Refunds are issued within 7 days. Refunds go to the original payment method."
answer = "Refunds are issued within 7 days. They go to the original payment method. Store credit is also offered."
STOP = {"are", "the", "to", "is", "also", "they", "within", "go"}
words = lambda s: {w for w in re.findall(r"[a-z0-9]+", s.lower()) if w not in STOP}
ctx = words(context)
claims = [s for s in re.split(r"(?<=\.)\s+", answer) if s]
supported = [c for c in claims if words(c) <= ctx]
print(f"\ngroundedness {len(supported)}/{len(claims)} = {len(supported) / len(claims):.2f}")
for c in claims:
print(" ", "ok " if c in supported else "NOT", c)
Output (Python 3.13.11):
retriever recall@3 recall@5 MRR nDCG@5
vector only 0.38 0.62 0.42 0.41
hybrid+rerank 1.00 1.00 0.88 0.91
groundedness 2/3 = 0.67
ok Refunds are issued within 7 days.
ok They go to the original payment method.
NOT Store credit is also offered.
The retrieval table is made-up data, but the shape is typical of what I’d look for. The vector-only retriever never surfaces c11 for q4 at all, which no prompt change can fix. The second retriever finds everything in the top 5, and MRR says it usually ranks the answer first. That gap is the argument for hybrid search and a reranker, made with numbers instead of opinion.
How do I measure faithfulness and answer quality?
Generation metrics need judgment, so most teams use an LLM as the judge. The Ragas faithfulness metric is a good template: break the answer into claims, check each claim against the retrieved context, and score the number of supported claims divided by the total, from 0 to 1. A faithful answer is one where every claim can be supported by the context.
My script does a lexical version of the same thing. It splits the answer into sentences and calls a sentence supported only if all its content words appear in the context. It flags “Store credit is also offered” because nothing in the context says so. That is the textbook RAG hallucination: fluent, plausible, unsupported. A word-overlap check is far too crude for production (it will fail paraphrases and pass negations), which is why the real version asks a model. LLM-as-a-judge covers how to make those judges reliable.
The other generation metrics worth tracking:
- Answer relevance: does the answer address the question that was asked? A faithful answer about the wrong thing still fails the user.
- Context recall against a reference: Ragas also defines context recall as the share of claims in a reference answer that the retrieved context supports. It needs a reference answer, but it measures retrieval in terms of facts rather than chunk IDs.
- Correctness against a reference answer, when you have one. For extractive questions (dates, amounts, names) exact match or a normalized string match is cheaper and more reliable than any judge.
- Refusal behavior: when the context lacks the answer, does the system say so? Include questions with no answer in the corpus on purpose.
How do I build a RAG test set?
This is the part people skip, and every metric above depends on it.
- Start from real questions. Pull them from search logs, support tickets or user interviews. Synthetic questions generated from your chunks are useful for volume, but they tend to reuse the chunk’s own wording, which flatters keyword and vector search alike.
- Label the evidence, not only the answer. For each question, record which chunk IDs contain the answer. Without that you can only measure end to end.
- Cover the failure types. Lookups, multi-hop questions, questions about recent changes, questions with no answer in the corpus, and questions phrased in different vocabulary from the docs.
- Keep labels stable across re-chunking. If chunk IDs change every time you change chunk size, map labels to document spans or answer strings instead.
- Version it. The test set is an asset. Store it next to the code and review changes to it like code.
A few hundred labeled questions is a reasonable target to start. For comparison, ARES reports needing only a few hundred human annotations to correct its automated judges. The right number for you depends on how many question types you need to cover, since each slice needs enough questions to show a change.
Where does RAG evaluation go wrong?
Judging with the same model that generates. A model tends to agree with its own style of answer. Use a different model or a stronger one for judging, and check its verdicts against human labels on a sample.
Averages hide the important failures. A 0.9 faithfulness average can include the five questions your legal team cares about most. Slice results by question type.
Measuring once. Embedding model upgrades, re-chunking, a new reranker and prompt edits all move the numbers. The test set has to run in CI or on a schedule, like any regression suite.
Ignoring cost and latency. A pipeline that improves recall@5 by retrieving 50 chunks and reranking them may double your latency. Put tokens per answer and p95 latency in the same report as quality.
What I would do
From a data-platform seat, the line I would circle is that a RAG eval is a data quality job before it is an AI job. Labeled question-to-chunk pairs, stored and versioned, run on every change, with results tracked over time. Most of the work is the dataset, not the metric code.
Concretely: I’d label 100 to 200 real questions with their evidence chunks, compute recall@k and MRR on every retrieval change, and run an LLM faithfulness judge on the final answers weekly and before releases. I’d add the no-answer questions on day one, because a RAG system that invents answers when the corpus is silent does more damage than one that retrieves badly. When the system grows a loop, as in agentic RAG, I’d add the step-level checks from how to evaluate AI agents.
FAQ
What are the main RAG evaluation metrics?
For retrieval: recall@k, precision@k, MRR and nDCG. For generation: faithfulness (are the answer’s claims supported by the context?), answer relevance and correctness against a reference answer.
What is faithfulness in RAG?
The share of claims in an answer that the retrieved context supports. Ragas computes it as supported claims divided by total claims, on a 0 to 1 scale.
Can I evaluate RAG without ground truth?
Partly. Ragas and ARES score faithfulness and relevance with LLM judges and no reference answers. Retrieval recall still needs labels of which chunks are relevant, and some human labels are needed to check the judges.
What is the difference between recall@k and precision@k?
Recall@k asks how many of the relevant chunks made the top k. Precision@k asks how many of the top k are relevant. Recall protects answer quality; precision protects the prompt from noise and cost.
Sources
- Es et al., Ragas: Automated Evaluation of Retrieval Augmented Generation, arXiv, 26 September 2023
- Saad-Falcon et al., ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems, arXiv, 16 November 2023
- Ragas docs, Faithfulness and Context recall (read 2026-10-31)