Most teams treat an agent's tool definitions as configuration: edit the JSON schema, redeploy, move on. That works until a change lands while conversations are in flight, or until a rename quietly degrades the model's behavior in ways no unit test catches. A tool schema is an API contract, and the client reading it is a model that has no changelog to consult.

What actually breaks when a schema changes

There are three distinct failure surfaces, and they need different defenses.

In-flight conversations. If a session has already shown the model a tool called search_orders with a customer_id argument, and your deploy replaces it with find_orders taking account_id, the history still contains calls to the old shape. The model may imitate its own earlier calls, and your executor rejects them. Conversation history is effectively a cached copy of your old API.

Stored traces and fixtures. Replay tests, golden trajectories and recorded calls all encode argument names. A rename invalidates them silently or loudly, depending on how strict your replay harness is.

Model behavior. Even a purely cosmetic change, like rewording a description or reordering parameters, shifts how often the model picks the tool and how it fills arguments. You should treat description edits as behavior changes and run your trajectory evals on them, not only on code changes.

A migration pattern that works

Borrow from ordinary API evolution, with a few adjustments for model clients:

  1. Prefer additive changes. Add an optional parameter rather than renaming one. Default it to the old behavior.
  2. Accept both shapes in the executor. When you must rename, let the executor accept the old and new argument names for a deprecation window, normalizing to the new form before any logic runs. Log which shape arrived so you know when the old one is dead.
  3. Version by name only for real semantic breaks. If meaning changes (say, a tool that returned a list now returns a page with a cursor), expose list_orders_v2 alongside the old tool, and remove the old one only after sessions drain or after a bounded session lifetime.
  4. Pin the tool set per session. Record a tool-set hash when a session starts and keep serving that set for that session. New sessions get the new set. This removes the in-flight mismatch entirely, at the cost of running two versions for a while.
  5. Return actionable errors for stale calls. If an old call does arrive, the error should say exactly what the current shape is, so the model can correct itself on the next turn.

Make it checkable

Keep tool schemas in version control as the single source of truth, generate the model-facing definitions from them, and add a CI check that diffs the schema against the previous release and fails on any removed or renamed field without a matching compatibility shim. Pair that with a small trajectory suite so description-only edits get exercised.

The closing rule is simple: the model's view of your tools is a published interface. Change it with the same discipline you would apply to a public endpoint, and the surprises move from production to the pull request.

AI AgentsSoftware Engineering