Most retrieval pipelines are built in one shot: embed the query, fetch the top few chunks, paste them into the prompt. That first stage is designed to be fast and to cast a wide net. It compares a query vector with precomputed document vectors, so it never reads the query and a candidate passage *together*. That is exactly why it is cheap, and exactly why it sometimes ranks a loosely related chunk above the one that actually answers the question.

A reranker is a second, slower pass over a small candidate set. It reads the query and each candidate jointly and produces a fresh relevance score. The common shape is: retrieve a generous candidate list (say 30 to 100), rerank it, and keep only the top handful for the prompt.

What the extra pass buys you

Reranking helps most when the right passage is *in* your candidate list but not near the top. Two symptoms point to this:

  • Answers are sometimes wrong even though a manual search shows the correct chunk was retrieved at rank 12 or 20.
  • You raised the number of chunks in the prompt to compensate, and cost went up while answer quality stayed flat or got worse, because more irrelevant text now competes for the model's attention.

If instead the right passage is never in the candidates, a reranker cannot help. That is a recall problem, and it belongs to chunking, query rewriting, or hybrid search. Reranking cannot rescue what was never fetched.

How to decide, with a measurement

You do not need a large study. Take a set of real questions where you know which passage should support the answer, and compute two numbers over your first-stage results:

  1. Recall at K_wide, where K_wide is your candidate size. This is your ceiling; no reranker exceeds it.
  2. Hit rate at K_final, the fraction of questions where the right passage is in the few chunks you actually send to the model, with and without reranking.

If the gap between those two numbers is large, a reranker has room to work. If they are already close, skip it and save the latency.

Then price the cost honestly. A reranker adds a model call whose time grows with candidate count and passage length, and it adds a dependency that can fail or time out. Decide up front what happens then: the safe default is to fall back to the first-stage order rather than fail the request.

A concrete closing rule

Treat reranking as a hypothesis, not a default. Measure the gap between candidate recall and final-slice hit rate on your own questions, add the pass only if that gap is meaningful, cap the candidate count so latency stays bounded, and always keep the first-stage ranking as the fallback. A pipeline that can explain *why* it has a reranker is easier to maintain than one that has it because a tutorial did.

AI IntegrationLarge Language Models