Where You Put an Instruction Matters As Much As What It Says

Where You Put an Instruction Matters As Much As What It Says

Most prompt debugging starts with wording: is the instruction clear, is the example good, is the tone right. But once a prompt gets long — a system prompt plus retrieved documents plus conversation history plus tool output — a second variable starts to dominate: *position*. Models are measurably better at attending to information near the beginning and end of a context window than to information buried in the middle. This isn't a quirk of one vendor's model; it shows up across architectures and has been documented widely enough that practitioners now just call it "lost in the middle." Treating it as a known engineering constraint, rather than rediscovering it every time a feature regresses, saves a lot of debugging time.

Where the effect actually bites

The failure mode rarely looks like a prompting bug. It looks like a feature that "used to work" and now doesn't, right after someone added another retrieved chunk, another few turns of conversation history, or another paragraph of tool output ahead of the actual instruction. A few concrete places this shows up:

• RAG pipelines that stuff five or six retrieved passages into context — the one that actually answers the question is rarely first or last.
• Agent scratchpads that accumulate turn after turn, quietly pushing the original task description further from the point where the model has to act.
• System prompts that get long enough that the one non-negotiable constraint (don't reveal X, always format as Y) sits in paragraph four instead of the last line before the user's request.
• Tool output dumped wholesale into context, with the instruction on what to *do* with that output appearing before the output itself, separated by thousands of tokens the model has to look past.

Practical mitigations

None of these require a different model — they're structural:

• Bookend critical instructions. State the non-negotiable constraint near the top and restate it immediately before the point of generation. Redundancy here is cheap; a missed constraint is not.
• Keep the ask close to the boundary. Whatever the model needs to act on right now belongs as close as possible to the end of the prompt, not buried under history it can skim past.
• Prune, don't just accumulate. Summarize stale turns instead of carrying full transcripts forward indefinitely — every additional token of dead weight pushes something else further from the edges.
• Test for position, not just content. Build a few eval cases where the correct answer is deliberately placed mid-context, and watch whether accuracy drops compared to the same content near the edges. If it does, you have a measurable regression to fix, not a vague feeling.
• Use structure to do the finding for the model. Clear headers and delimiters around the relevant section reduce how much the model has to rely on position alone to locate what matters.

The takeaway for prompt review: don't just ask "does this instruction say the right thing." Ask "does this instruction survive being the fifteenth paragraph instead of the first." If the answer is no, the fix isn't a smarter model — it's better placement.