---
name: rag-eval-design
source: https://app.decimal.ai/s/rag-eval-design@1/SKILL.md
source_sha256: 8e26a2b85aba
---

# RAG / retrieval eval design

## What this does

Designs an evaluation for a system that **retrieves** context (chunks, passages, documents)
and then **generates** an answer from it. The core discipline a generic eval skips: score the
two stages **separately**, always include a **groundedness** check, and build a labeled golden
set that has **hard negatives** — not one blended "does it answer well?" number.

## When to use / when NOT to

**Use when** someone is testing, benchmarking, or building an eval / test set for a
document-Q&A bot, knowledge-base assistant, support-answer bot over articles, or any
retrieve-then-answer pipeline — and wants to know how to measure it.

**Do NOT use for:**
- General eval-function or scoring-rubric authoring on non-retrieval tasks — that is
  `eval-writer` / `prompt-eval-rubric-writer`.
- Making an agent answer strictly from provided sources at runtime — that is
  `rag-grounding-and-citation`. This skill designs the *eval*, it does not do the answering.

## Rule 1 — Score retrieval and generation as two separate stages

A retrieve-then-answer system has two independent failure surfaces. Collapsing them into one
score hides *which* stage broke, and the two need different fixes.

- **Retrieval metrics** grade the retriever **alone**, against labeled relevant chunks,
  independent of the generator. The generator can never answer from a chunk that was never
  retrieved. Measure:
  - **recall@k** — did a chunk that actually answers the question land in the top *k*?
  - **precision@k** — of the top *k*, how many are actually relevant?
  - **MRR** / **hit-rate** — how high did the first relevant chunk rank?
- **Generation metrics** grade the answer **given** the retrieved context:
  - **groundedness / faithfulness** — see Rule 2.
  - **answer relevance** — does the answer actually address the question asked?
  - **completeness** — does it use the relevant information that *was* retrieved?

Choose *k* to match the generator's context budget: measuring recall@20 is meaningless if you
only feed the top 5 chunks to the model.

## Rule 2 — Groundedness is mandatory, and it is not accuracy

**Faithfulness / groundedness = every claim in the answer is supported by the retrieved
context.** Grade it claim-by-claim against the retrieved chunks — *not* against world truth.

An answer can be factually correct (pulled from the model's training memory) and still
**unfaithful** because the retrieved context does not support it. A generic "is this a good
answer?" judge rewards that answer; a grounded system must not. Every RAG eval needs an
explicit groundedness dimension, scored against the retrieved passages, or it is not a RAG eval.

## Rule 3 — The golden set must have hard negatives

Label each question with:
- the chunk(s) that actually answer it (ground truth for retrieval metrics), and
- a reference answer (for generation).

Then deliberately add **hard negatives**, or the numbers lie:
- **Out-of-corpus questions** — the answer is *not* anywhere in the documents. The correct
  behavior is to refuse ("not in the sources"). This is the only way to measure refusal and
  false-answer rate. A test set of only answerable questions never catches a system that makes
  things up on the ones it can't answer.
- **Distractor passages** — chunks that are topically similar but do *not* answer the question.
  Without them, precision is untested and a retriever that returns everything scores perfect
  recall while teaching you nothing.

## Rule 4 — Test the RAG-specific failure modes by name

Include a labeled case for each, because they have different root causes and the separated
metrics are what let you tell them apart:

| Failure mode | Signature in the metrics | Where the fix lives |
|---|---|---|
| **Retrieval miss** | low recall@k — the answering chunk never made the top *k* | retriever: embeddings, chunking, *k* |
| **Hallucination despite context** | low faithfulness while retrieval recall is high | generator / grounding prompt |
| **Ignoring retrieved context** | correct chunk WAS retrieved, yet the answer contradicts it or falls back to training | generator |

The diagnostic power comes from Rule 1: **high recall + low faithfulness = a generation bug;
low recall = a retrieval bug.** A single blended score can localize neither.

## How to assemble the eval

1. **Build the labeled golden set** — real questions; for each, the answering chunk id(s) and a
   reference answer. Mix in out-of-corpus questions and distractor chunks (Rule 3).
2. **Log intermediate outputs** — capture the *retrieved chunks*, not just the final answer.
   You cannot compute retrieval metrics or grade faithfulness against context you didn't record.
3. **Score retrieval** from the logged chunks vs. the labels: recall@k, precision@k, MRR/hit-rate.
4. **Score generation** given those chunks: groundedness (claim-by-claim vs. retrieved text),
   answer relevance, completeness; plus refusal-correctness on the out-of-corpus questions.
5. **Report the two stages separately**, then read the failure table (Rule 4) to localize.

## Worked example

System: a bot that embeds internal PDFs, retrieves the top 5 chunks, and writes an answer.

- **Golden set:** 60 questions. 45 answerable (each tagged with its answering chunk + a
  reference answer), 10 out-of-corpus (answer: refuse), 5 with planted distractor chunks.
- **Retrieval:** recall@5 = 0.82, MRR = 0.71 → 18% of questions never surface the right chunk.
- **Generation (on the 82% with a good chunk):** faithfulness = 0.9, relevance = 0.95.
- **Refusal (10 out-of-corpus):** answered anyway 4/10 → hallucination problem on the tail.

Read-out: retrieval is the bigger miss (fix chunking/embeddings); generation is faithful when
fed a good chunk but invents answers when the corpus lacks one (tighten the refusal behavior).
A single "0.8 answer quality" score would have surfaced none of this.

## Common mistakes (this is the lift)

1. **One blended score** ("rate the answer 1–5") that conflates retrieval and generation, so a
   failure can't be localized.
2. **No groundedness dimension** — grading answers against world truth instead of against the
   retrieved context, letting confident-but-unsupported answers pass.
3. **A golden set of only answerable questions** — never measuring refusal or false-answer rate.
4. **No distractors** — precision untested; a return-everything retriever looks perfect.
5. **Grading only the final answer** — never logging retrieved chunks, so retrieval metrics and
   faithfulness are uncomputable.
6. **recall@k with a k that doesn't match** what actually gets fed to the generator.

## References

- `references/retrieval-metrics.md` — exact formulas and when to use each retrieval metric
  (recall@k, precision@k, MRR, hit-rate, nDCG), micro-vs-macro averaging, and choosing *k*.