RAG / retrieval eval design

What this does

Designs an evaluation for a system that retrieves context (chunks, passages, documents) and then generates an answer from it. The core discipline a generic eval skips: score the two stages separately, always include a groundedness check, and build a labeled golden set that has hard negatives — not one blended "does it answer well?" number.

When to use / when NOT to

Use when someone is testing, benchmarking, or building an eval / test set for a document-Q&A bot, knowledge-base assistant, support-answer bot over articles, or any retrieve-then-answer pipeline — and wants to know how to measure it.

Do NOT use for:

• General eval-function or scoring-rubric authoring on non-retrieval tasks — that is

eval-writer / prompt-eval-rubric-writer.

• Making an agent answer strictly from provided sources at runtime — that is

rag-grounding-and-citation. This skill designs the eval, it does not do the answering.

Rule 1 — Score retrieval and generation as two separate stages

A retrieve-then-answer system has two independent failure surfaces. Collapsing them into one score hides which stage broke, and the two need different fixes.

• Retrieval metrics grade the retriever alone, against labeled relevant chunks,

independent of the generator. The generator can never answer from a chunk that was never retrieved. Measure:

• recall@k — did a chunk that actually answers the question land in the top k?
• precision@k — of the top k, how many are actually relevant?
• MRR / hit-rate — how high did the first relevant chunk rank?
• Generation metrics grade the answer given the retrieved context:
• groundedness / faithfulness — see Rule 2.
• answer relevance — does the answer actually address the question asked?
• completeness — does it use the relevant information that was retrieved?

Choose k to match the generator's context budget: measuring recall@20 is meaningless if you only feed the top 5 chunks to the model.

Rule 2 — Groundedness is mandatory, and it is not accuracy

Faithfulness / groundedness = every claim in the answer is supported by the retrieved context. Grade it claim-by-claim against the retrieved chunks — not against world truth.

An answer can be factually correct (pulled from the model's training memory) and still unfaithful because the retrieved context does not support it. A generic "is this a good answer?" judge rewards that answer; a grounded system must not. Every RAG eval needs an explicit groundedness dimension, scored against the retrieved passages, or it is not a RAG eval.

Rule 3 — The golden set must have hard negatives

Label each question with:

• the chunk(s) that actually answer it (ground truth for retrieval metrics), and
• a reference answer (for generation).

Then deliberately add hard negatives, or the numbers lie:

• Out-of-corpus questions — the answer is not anywhere in the documents. The correct

behavior is to refuse ("not in the sources"). This is the only way to measure refusal and false-answer rate. A test set of only answerable questions never catches a system that makes things up on the ones it can't answer.

• Distractor passages — chunks that are topically similar but do not answer the question.

Without them, precision is untested and a retriever that returns everything scores perfect recall while teaching you nothing.

Rule 4 — Test the RAG-specific failure modes by name

Include a labeled case for each, because they have different root causes and the separated metrics are what let you tell them apart:

| Failure mode | Signature in the metrics | Where the fix lives | |---|---|---| | Retrieval miss | low recall@k — the answering chunk never made the top k | retriever: embeddings, chunking, k | | Hallucination despite context | low faithfulness while retrieval recall is high | generator / grounding prompt | | Ignoring retrieved context | correct chunk WAS retrieved, yet the answer contradicts it or falls back to training | generator |

The diagnostic power comes from Rule 1: high recall + low faithfulness = a generation bug; low recall = a retrieval bug. A single blended score can localize neither.

How to assemble the eval

1. Build the labeled golden set — real questions; for each, the answering chunk id(s) and a

reference answer. Mix in out-of-corpus questions and distractor chunks (Rule 3).

1. Log intermediate outputs — capture the retrieved chunks, not just the final answer.

You cannot compute retrieval metrics or grade faithfulness against context you didn't record.

1. Score retrieval from the logged chunks vs. the labels: recall@k, precision@k, MRR/hit-rate.
2. Score generation given those chunks: groundedness (claim-by-claim vs. retrieved text),

answer relevance, completeness; plus refusal-correctness on the out-of-corpus questions.

1. Report the two stages separately, then read the failure table (Rule 4) to localize.

Worked example

System: a bot that embeds internal PDFs, retrieves the top 5 chunks, and writes an answer.

• Golden set: 60 questions. 45 answerable (each tagged with its answering chunk + a

reference answer), 10 out-of-corpus (answer: refuse), 5 with planted distractor chunks.

• Retrieval: recall@5 = 0.82, MRR = 0.71 → 18% of questions never surface the right chunk.
• Generation (on the 82% with a good chunk): faithfulness = 0.9, relevance = 0.95.
• Refusal (10 out-of-corpus): answered anyway 4/10 → hallucination problem on the tail.

Read-out: retrieval is the bigger miss (fix chunking/embeddings); generation is faithful when fed a good chunk but invents answers when the corpus lacks one (tighten the refusal behavior). A single "0.8 answer quality" score would have surfaced none of this.

Common mistakes (this is the lift)

1. One blended score ("rate the answer 1–5") that conflates retrieval and generation, so a

failure can't be localized.

1. No groundedness dimension — grading answers against world truth instead of against the

retrieved context, letting confident-but-unsupported answers pass.

1. A golden set of only answerable questions — never measuring refusal or false-answer rate.
2. No distractors — precision untested; a return-everything retriever looks perfect.
3. Grading only the final answer — never logging retrieved chunks, so retrieval metrics and

faithfulness are uncomputable.

1. recall@k with a k that doesn't match what actually gets fed to the generator.

References

• references/retrieval-metrics.md — exact formulas and when to use each retrieval metric

(recall@k, precision@k, MRR, hit-rate, nDCG), micro-vs-macro averaging, and choosing k.