When a model asks for several tool calls in one turn, the happy path is easy: run them concurrently, collect the results, send them back. The interesting question is what happens when three of five calls succeed, one times out, and one returns an error. Most agent code answers that question by accident, and the accident is usually one of two bad defaults: fail the whole batch, or silently drop the failures.

Decide what a partial batch means

Start by classifying the calls in a batch by how they relate to each other:

  • Independent reads. Five lookups that don't depend on each other. A partial result is still useful, and the model can often proceed with what it has.
  • Dependent reads. The answer only makes sense with all inputs, such as comparing two prices. A missing half makes the other half misleading.
  • Writes. Side effects that may or may not have happened. A timeout here is not a failure; it is an unknown.

Each class needs a different policy. Independent reads can return partial results. Dependent reads should fail as a unit or be retried before anything goes back to the model. Writes should never be retried blindly, which is where idempotency keys earn their place.

Report every call, in order, with an explicit status

The worst thing you can do is return only the successes. The model asked for five things and will assume it got five answers, or quietly invent the missing ones. Return one result per requested call, matched by call ID, each with a status:

[
  {"call_id": "a1", "status": "ok", "result": {...}},
  {"call_id": "a2", "status": "error", "retryable": true, "message": "upstream timed out after 5s"},
  {"call_id": "a3", "status": "unknown", "message": "request sent; no response received"}
]

Note the third status. "Unknown" is distinct from "error" because the correct next action differs: for an error you may retry; for unknown you must check state first. Collapsing them is how an agent ends up sending the same email twice.

Bound the batch, and the wait

Run concurrent calls with a shared deadline rather than waiting on the slowest one indefinitely. When the deadline passes, cancel what you can, mark the rest as unknown, and return. Also cap how many calls you will execute per turn. A model that requests forty calls is usually confused, and executing all forty hides the confusion.

Test the policy, not just the calls

Write tests that inject a failure into each position of a batch: first, middle, last, and all of them. Assert on the invariants rather than exact output: every requested call ID appears in the response exactly once, writes with unknown status are never retried without a state check, and the model-facing message names which calls failed.

The closing point: a tool-calling loop is a small distributed system, and a partial failure is its normal operating condition. Write the failure policy down before the first production incident writes it for you.

AI AgentsAI Coding Assistants