Context Budgeting: A Practical Framework for Deciding What Actually Fits in the Window

Context Budgeting: A Practical Framework for Deciding What Actually Fits in the Window

Most teams treat the context window as a single pool: keep adding retrieved chunks, conversation turns, and tool output until the request stops erroring. That works until it doesn't — usually right when a feature gets popular enough to have long conversations, large tool outputs, or both at once. The fix isn't a bigger model. It's treating the window like a budget with named line items, the same way you'd budget memory in an embedded system, instead of a bucket you pour things into until it overflows.

Start by splitting the window into fixed and variable allocations. System instructions, output format rules, and few-shot examples are fixed costs — they don't grow with usage, so measure them once and subtract them off the top. Everything else — retrieved documents, conversation history, tool results — is variable, and it's the variable part that needs an actual policy, not "whatever fits." Without one, the loudest or most recent input wins by default, which is rarely the input that matters most.

Rank before you truncate

The mistake most truncation logic makes is cutting by position — dropping the oldest turns, or the last chunks retrieved — instead of by relevance. Position is a proxy for importance, and it's a bad one: a decision made three turns ago can matter more than the user's last sentence, and a lower-ranked retrieval result can be more relevant to the actual question than one that merely scored higher on keyword overlap. Score every candidate for inclusion — retrieved chunks by relevance, conversation turns by whether they contain unresolved decisions or constraints, tool output by whether downstream steps reference it — and fill the budget in ranked order, not chronological order. This also means your retrieval and history systems need to expose a relevance score at all, not just a ranked list; if they don't, that's the first gap to close.

Reserve tokens for output before you fill the input side. It's tempting to stuff the window with input and hope the model's response fits in whatever's left, but a truncated response is worse than a slightly smaller input — it fails visibly instead of degrading gracefully. Fix the output reservation as a hard constant and treat it as non-negotiable, the same way you'd reserve headroom for a stack in a low-memory environment.

Finally, log the budget breakdown per request, not just total tokens. When quality degrades, "average tokens per call" tells you nothing about why. A per-request log of how many tokens went to system instructions, history, retrieval, and tool output — plus what got dropped and why — turns a vague "the model seems worse lately" complaint into a specific, fixable claim: retrieval is crowding out history, or history is crowding out the one tool result that actually answers the question. Context budgeting isn't a one-time design decision; it's an operational discipline that needs the same visibility you'd give any other resource under contention.