Agents are happy to call a tool that returns a 4 MB JSON blob, and the model will dutifully try to read all of it. The result is predictable: the context fills with data the model will never use, cost and latency jump, and the details that mattered get buried. The tool worked perfectly. The design around it did not.

The fix is not a bigger context window. It is deciding, in the tool layer, how much of a result the model is allowed to see.

Treat the tool result as a view, not a dump

A tool's job is to return what the agent needs for its next decision, not everything the backend knows. Three techniques cover most cases:

  • Project fields. If an API returns 60 fields per record and the agent only reasons about five, return five. Make the field list part of the tool's contract rather than an accident of the upstream response.
  • Paginate with a handle. Return the first N rows plus a cursor and the total count, for example {"rows": [...], "returned": 25, "total": 8312, "next_cursor": "abc"}. The total matters: it tells the model the result is incomplete, so it does not conclude that 25 rows is everything.
  • Summarize deterministically. For large logs or documents, return counts, top-K groups, and the first and last few lines, not an LLM-generated summary. Deterministic summaries are testable and do not add a second source of hallucination.

Truncate loudly, never silently

The worst failure mode is a tool that quietly cuts off output at some limit. The model treats what it sees as complete and reasons confidently from a partial picture. If you truncate, say so inside the result, in words the model will act on:

[truncated: showing 50 of 1,204 matching lines. Narrow the query
 (add a date range or a path filter) or request the next page.]

That message does two things: it prevents false completeness, and it nudges the agent toward a better next call instead of retrying the same one. Pair it with a hard byte cap enforced in code so no single tool can consume more than a fixed share of your context budget regardless of what upstream returns.

Offer a narrowing path

Truncation only helps if the agent can get to the part it actually needs. Give large-result tools filter parameters (time ranges, IDs, substring match) and consider a second, cheaper tool for inspection, such as count_matches or list_columns, so the agent can look before it fetches. An agent that can ask "how big is this?" makes much better decisions than one that has to fetch to find out.

How to verify it

Write a test that feeds each tool an adversarially large upstream response and asserts three things: the output stays under the byte cap, the truncation notice is present, and the cursor or filter instructions are valid. Then replay a few real agent traces and check whether any run reached a wrong conclusion from a truncated result.

The takeaway: the size and shape of a tool's output is part of its interface design. Decide it deliberately, enforce it in code, and make every omission visible to the model.

AI AgentsSoftware Engineering