Most retrieval-augmented pipelines start the same way: embed the documents, embed the query, take the nearest neighbours by similarity, and call it retrieval. It works well enough in a demo to feel finished. Then someone searches for an exact error code, a product SKU, a version number or a person's name, and the results come back semantically related but wrong: passages "about" the right topic, missing the one that contains the string the user typed.

That isn't a tuning problem. It's a structural gap in what embedding similarity is good at, and it's the reason most production search systems combine two retrieval methods.

Two retrievers, two different blind spots

Vector (dense) search finds information that is conceptually similar to the query even when no keywords match. It will connect "how do I cancel my subscription" to a page titled "ending your plan."

Keyword (full-text) search, usually ranked with BM25, is precise on exact terms. Microsoft's documentation on hybrid search says it directly: scenarios such as querying over product codes, highly specialized jargon, dates and people's names perform better with keyword search because it can identify exact matches.

Each is blind where the other sees. That's why combining them isn't a hedge; it covers two different failure modes.

How hybrid search works

In Azure AI Search's description, hybrid search is a single query that runs a full-text search and a vector search in parallel, then merges the two result lists with Reciprocal Rank Fusion (RRF). The two retrievers score on very different scales (BM25 relevance versus vector distance), so averaging raw scores is fragile. RRF sidesteps that by working from ranks: each document is scored by summing 1 / (k + rank) across the lists it appears in, where k is a smoothing constant. A document ranked well by both retrievers rises to the top, and one retriever returning nothing useful doesn't ruin the result. There is no blend weight to tune.

Anthropic's Contextual Retrieval write-up reaches the same conclusion from the retrieval-quality side: embeddings capture semantic relationships but miss exact matches, BM25 finds precise terms, and combining them balances precise term matching with broader semantic understanding. In their evaluation, adding BM25 on top of contextual embeddings reduced the failed-retrieval rate further, from 3.7% to 2.9%. That's the vendor's own test, on its own datasets, so use it as direction, not as a promise.

The second lever: reranking

Retrieval's job is to produce a manageable candidate set with high recall, say the top 50. A reranker's job is to reorder those candidates by reading the query and each candidate together, which is more accurate than comparing precomputed vectors but too slow to run over a whole corpus. The pattern is retrieve broad and cheap, rerank narrow and expensive. In the same Anthropic evaluation, adding reranking brought the failed-retrieval rate down again, to 1.9%.

Azure's hybrid query also shows the pieces fitting together: full-text and vector queries merged by RRF, then an optional semantic ranker applied on top.

How to tell whether you need it

Hybrid search is not a day-one requirement for every RAG feature. If your corpus is homogeneous prose with no identifiers, codes or proper nouns that users search for verbatim, pure vector search may never show the gap.

The diagnostic is specific. Pull a sample of queries that returned bad results and check whether the correct passage contained a token with no natural paraphrase: an ID, a name, an acronym, an error code. If that pattern keeps showing up, a better embedding model or more chunk overlap won't fix it, because the problem isn't semantic. It's exact-match, and dense-retrieval tuning can't solve an exact-match problem.

What to do

  1. Build a small test set from real queries with known correct passages, and measure whether the right passage appears in the top results. Don't judge by whether the final answer sounds plausible.
  2. Add keyword search alongside vector search, merged with RRF, and compare on that test set.
  3. Add a reranker if recall looks fine but the right passage sits too low in the list.
  4. Re-measure after every change. Gains vary by data, so keep the test set and rerun it.

For the pre-retrieval half of the problem, see how to split documents for RAG, and for what happens after retrieval, why passage order changes the answer.

Sources

Developer ToolsLarge Language ModelsSoftware Engineering