Most agent tests look at the last message: does the answer contain the right order number, does the summary mention the right file? That's necessary, but it leaves a blind spot. An agent that reaches the right answer after 14 tool calls, two of them redundant and one a write it should never have attempted, passes the same test as a clean three-call run.

Testing the path is tempting, and it's also easy to get wrong. Anthropic's engineering guide "Demystifying evals for AI agents" gives a balanced view, and this article follows it.

Two things to look at: the transcript and the outcome

The guide separates a transcript, the full record of what the agent did, including tool calls and reasoning, from the outcome, the actual final state of the environment. Its example: a flight-booking agent's transcript may say "Your flight has been booked," but the outcome that matters is whether a reservation exists in the database.

So the first rule is to check the environment, not the agent's claim about it. Did the refund row get created? Did the file change? That's an outcome test, and it's the most important one.

A warning about over-specifying the path

Anthropic is direct about a common instinct: checking that an agent followed very specific steps, such as a particular sequence of tool calls in a particular order. The guide says it found this approach too rigid and that it produces overly brittle tests. Its recommendation is to grade what the agent produced, not the path it took.

That's a useful correction to the idea that agent tests should mirror an exact transcript. Models legitimately reach the same result by different routes, and a test that insists on one route fails for no good reason.

What's worth asserting about the path

The way through is to assert invariants, rules that must hold on every acceptable path, instead of an exact sequence. These are my suggestions, in order of value:

  • Forbidden calls never happen. A read-only task should produce zero write-capable tool calls. This is the cheapest and most valuable trajectory assertion, and it doesn't constrain how the agent solves the task.
  • Argument sanity. The refund amount must not exceed the order total, and the file path must stay inside the allowed directory. Check these with ordinary code, not another model.
  • Budgets. Cap the number of calls and tokens. A run that suddenly needs three times the steps is a regression even if the answer is right.
  • A precondition, not a script. If a refund must be preceded by a lookup of that order, assert that precondition, but don't pin the other steps around it.

Pick the right grader

The guide describes three kinds:

  • Code-based graders (string matches, tests, static analysis, tool-call checks): fast, cheap, objective and reproducible, but brittle to valid variations.
  • Model-based graders (rubrics, natural-language assertions): flexible and good at nuance, but non-deterministic and costlier.
  • Human graders: the gold standard for quality, but expensive and slow.

Use code for the invariants above, a model grader for qualities such as helpfulness, and humans to calibrate both. For more on model graders, see pairwise judging and position bias.

Handle non-determinism with the right metric

Agents vary between runs, so one pass proves little. The guide distinguishes two metrics:

  • pass@k: the probability of at least one success in k attempts.
  • pass^k: the probability that all k attempts succeed, a measure of reliability.

Its example: with a 75% per-trial success rate, the chance of passing three trials in a row is about 42%. If your agent acts unattended, reliability is what matters, so a forbidden-call assertion should require zero violations across all runs.

How to build it

Record the trajectory as plain data, a list of {tool, args, result_summary} entries, using a thin wrapper around your tool dispatcher. Write assertions as ordinary test functions over that list and over the final environment state. Stub or sandbox the tools so tests can't touch real systems. Keep the trajectories of failing runs: the recorded path shows exactly where the agent diverged. For tool-level failures to include in these tests, see Designing Tool Error Messages an Agent Can Act On.

The takeaway

Test outcomes first, then add a few invariants about the path, and avoid scripting the exact steps. If you can afford only one new agent test, make it a forbidden-calls assertion on a read-only scenario. It takes minutes to write and guards against the failures that are most expensive to find in production.

Sources

AI AgentsSoftware Engineering