RAG teams spend most of their tuning budget on the retriever: better embeddings, better chunking, a reranker to float the most relevant passage to the top. All of that assumes that once a passage is in the prompt, the model uses it in proportion to how relevant it is. That assumption is shakier than it looks, and it's cheap to test directly.

What the research found

The "lost in the middle" paper (Liu et al., 2023) tested language models on two tasks that require finding information in a long input: multi-document question answering and key-value retrieval. Its central finding, in the authors' words, is that performance is often highest when the relevant information is at the beginning or end of the input context, and that performance significantly degrades when models must access relevant information in the middle of long contexts, even for models explicitly designed for long contexts. The authors conclude that models do not robustly use information spread throughout a long input.

Two caveats worth stating. The paper is from 2023 and models have changed since, so don't assume the exact size of the effect in the version you use. And it studied two specific tasks, so how it appears in your application is an empirical question. That's the reason to test.

Why it matters for RAG

In a RAG pipeline, the practical consequence is specific. A reranker can correctly identify the single best passage and the prompt-assembly step can still place it fourth out of seven, a position the model may use less reliably. The answer then looks exactly like a retrieval failure. Teams that don't check position end up retuning the retriever (new embeddings, different chunk sizes) when the defect is in how the retrieved set is assembled into the prompt.

A cheap test you can run this week

  1. Take a set of questions with known answers and a known supporting passage for each.
  2. Hold everything fixed: the retrieved set, the other passages, the prompt template.
  3. Vary only where the supporting passage sits: first, middle, last.
  4. Compare accuracy by position.

If accuracy changes meaningfully with position alone, you've confirmed the effect on your own stack instead of assuming a paper's result applies. If it doesn't, you've ruled out a failure mode and can stop worrying about it. Either result is useful.

If you do see the effect

These fixes are my suggestions, to be validated by the same test:

  • Reorder the assembled prompt. Try placing the highest-ranked passages at the edges of the context, near the start or near the question, rather than burying them in the middle. Measure which placement works best for your model, because the paper reports that the beginning and end both do well and does not say one always wins.
  • Use fewer passages. Every extra passage is another chance to bury the important one. When precision matters more than exhaustive recall, cut the list.
  • Give the reranker a second job. Optimize for the final position of passages in the assembled prompt, not just each one's relevance score in isolation.
  • Don't over-correct. More context isn't always worse. In Anthropic's Contextual Retrieval write-up, passing the top 20 chunks worked better than 5 or 10. The right number is something you measure, not something you assume. See how to split documents for RAG.

Make it a standing test

Treat context assembly as its own pipeline stage with a controlled input and a measurable output, the same way you'd test retrieval. Add the position test to your evaluation suite so it reruns when you change the model, the prompt template or the number of retrieved passages. Models change, and a position effect that was small last quarter can be larger after an upgrade. If you also run hybrid retrieval, run the position test on its output.

The takeaway

"Retrieved" and "used" are not the same thing. A passage can be the best match and still be underused because of where it sits. Test for it, because the fix is cheap and doesn't touch your retriever.

Sources

AI ModelsLarge Language ModelsSoftware Engineering