When a prompt is three lines long, ordering barely matters. When it carries tens of thousands of tokens of documents, logs, or retrieved passages, the position of each piece starts to affect results. Models do not attend uniformly across a long input, and the effect varies by model, so the honest advice is: pick a sensible default, then measure.

A defensible default layout

A structure that works well as a starting point:

  1. Role and stable rules at the top. These rarely change, and keeping them first also helps prompt caching.
  2. Reference material (documents, code, transcripts) in the middle, each item wrapped in clearly labelled delimiters with an identifier.
  3. The task and the specific question at the end, close to where the model begins generating.

Putting the question after the material means the model reads the documents already knowing nothing about your goal, but it answers with the question fresh in view. Many teams also restate the key constraint (output format, "answer only from the documents") right before the question, because instructions buried 30k tokens earlier are the ones that get dropped.

Make the material easy to address

Long inputs go wrong when the model cannot tell where one source ends and another begins. Give every document a stable tag and ask the model to reference those tags in its answer:

<document id="refund-policy-v3">...</document>
<document id="support-faq">...</document>

This improves citation checking and makes it far easier to debug which passage influenced a bad answer.

Test placement instead of arguing about it

Build a small harness that varies one thing at a time on your own eval set:

  • question before vs. after the documents
  • most relevant passage first, last, or in the middle
  • instruction repeated at the end vs. stated once

Run each variant over the same cases and compare. If the "needle" document position changes your accuracy, you have found a real sensitivity, and you can mitigate it by reranking so the best evidence sits where your model handles it best, or by sending fewer passages.

The takeaway

Treat ordering as a tunable parameter. Start with stable rules first, labelled material second, and the question plus a restated key constraint last, then let your eval set, not intuition, decide whether to move anything.

AI IntegrationSoftware Engineering