Grade Retrieval Before Generation: Recall at k as the First Check on a RAG System

When a RAG answer is wrong, the instinct is to edit the prompt. But a question-answering system is two systems stacked together: one that finds passages, and one that writes from them. If you only grade the final answer, you cannot tell which one failed, and you will spend days tuning the wrong half.

Grade retrieval without the generator in the loop

The retrieval question is simple: for this query, did the passages that contain the answer show up in what the model was given? You can answer it with no LLM calls at all.

Build a small set of queries, and for each one record which document or chunk IDs a correct answer needs. Then run only the retriever and compute two things:

• Recall at k: the fraction of queries where at least one needed chunk appears in the top k results you actually pass to the model. Use the same k as production, not a larger one that flatters the number.
• Rank of the first hit: where in the list the first needed chunk lands. Models attend unevenly across long contexts, so a needed chunk in position nine behaves differently from one in position one.

These checks are deterministic, fast and free, so you can run them on every change to chunking, embeddings, or query handling. That makes them far better regression tests than anything involving a judge model.

Then read the failures by type

Once you have the misses, sort them rather than counting them. In practice they fall into a few buckets:

1. The content is not in the corpus. No retrieval change fixes this. The right behavior is for the system to say it does not know, which is a generation test, not a retrieval one.
2. The content is there but split badly. The answer spans two chunks and neither one alone matches the query.
3. The wording differs. The user says "cancel", the document says "terminate the subscription".
4. It was retrieved but ranked too low to make the cut.

Each bucket has a different remedy: corpus work, chunking, query rewriting or hybrid search, and reranking respectively. Without the sort, every miss looks like "retrieval is bad".

Now grade generation on its own

To test the generator in isolation, hand it the known-good passages directly and see whether it answers correctly and stays within them. If it fails with perfect context, the prompt or model is the problem. If it succeeds with perfect context but fails in the full pipeline, you have localized the fault to retrieval, which is the cheaper half to fix.

What to do Monday

Write down twenty real user questions, mark the chunk IDs that answer each, and compute recall at your production k. If the number is below what you would accept, stop editing prompts. That single number tells you which half of the system deserves your next week.