Most teams evaluate a retrieval-augmented feature by reading the final answers and deciding whether they look right. When an answer is wrong, the cause could be that the right document was never retrieved, that it was retrieved but ranked too low to make the context, or that the model had the right material and still got it wrong. These are different bugs with different fixes, and an end-to-end judgment can't tell them apart. The fix is to evaluate the two halves separately.

Build a retrieval-only test set

Start with a list of real questions. For each one, record which document or passage a correct answer must draw on. You are labeling evidence, not answers. A useful set is often only 40 to 100 questions, and it can be assembled from support tickets, search logs, or by having a subject-matter expert write questions against documents they know well.

Label more than one acceptable source when several genuinely contain the answer. Otherwise you'll penalize retrieval for finding a perfectly good alternative.

Measure what retrieval actually controls

With labels in place, run retrieval alone, with no model involved, and compute:

  • Recall@k: for the k passages you actually put in the prompt, how often does at least one required passage appear? This is your ceiling. If recall@k is 70%, no prompt improvement can push answer quality above roughly 70% on those questions.
  • Rank of the first relevant hit: a passage at position 9 may exist in your results and still never fit in the context window after budgeting.
  • Precision-flavored checks: how much of what you retrieved is irrelevant? Noise costs tokens and can distract the model.

These run in seconds, cost nothing in model calls, and are deterministic, so you can run them on every change to chunking, embeddings, or query rewriting.

Then evaluate generation with retrieval held fixed

To test the generation half, feed the model the known-correct passages directly, bypassing retrieval. If answers are still poor, the problem is in the prompt, the model, or the task definition. If they're good, the problem lives upstream.

Comparing three numbers tells you where to spend effort:

  1. Answer quality with oracle passages (generation ceiling).
  2. Recall@k from retrieval alone (retrieval ceiling).
  3. End-to-end answer quality (what users see).

A large gap between 1 and 3 with high recall points to how context is assembled: ordering, truncation, or formatting. A low recall number means fix retrieval first, whatever the prompt looks like.

Keep the test set honest

Retrieval sets go stale as the corpus changes. When a document is renamed, split, or removed, labels silently break, so store stable identifiers and rerun the suite whenever the corpus is re-indexed. Add every retrieval failure you find in production as a new labeled question.

The takeaway

An end-to-end score tells you something is wrong. A split evaluation tells you which half to fix. Label evidence once, and you get a fast, cheap, deterministic regression test for the half of the system that engineers change most often.

Developer ToolsLarge Language ModelsSoftware Engineering