Ask an agent to "figure out the right next step" and it usually will, until the one time it doesn't, and you're left debugging a decision that exists only as a pattern the model inferred from a system prompt. A lot of agent flakiness isn't a capability problem. It's a control-flow problem: teams write state transitions in prose ("first check X, then if Y do Z, otherwise...") and expect a language model to enforce them as reliably as an if-statement.

Anthropic's engineering guide "Building effective agents" gives the vocabulary for this decision, and this article builds on it.

Workflows versus agents

The guide draws a line between two kinds of system. Workflows orchestrate LLMs and tools through predefined code paths. Agents let the LLM dynamically direct its own process and tool use. Its summary of the trade-off: workflows offer predictability and consistency for well-defined tasks, while agents are the better option when flexibility and model-driven decision-making are needed at scale. It also notes that agentic systems often trade latency and cost for better task performance, and advises finding the simplest solution possible and adding complexity only when needed.

Most "agent flakiness" comes from using an agent where a workflow would do, or from mixing the two without deciding which layer owns which decision.

The workflow patterns that replace prompted control flow

The guide describes several patterns that put the flow in code:

  • Prompt chaining: split a task into fixed sequential steps, each LLM call handling the previous output, with programmatic checks between steps. Use it when the task decomposes cleanly into fixed subtasks. It trades latency for accuracy.
  • Routing: classify the input and send it to a specialized follow-up. It works well when there are distinct categories better handled separately.
  • Parallelization: run independent subtasks at once (sectioning), or run the same task several times for confidence (voting), then aggregate in code.
  • Orchestrator-workers: a central LLM breaks down the task and delegates, for when you can't predict the subtasks in advance. That's the pattern behind multi-agent systems.
  • Evaluator-optimizer: one model generates, another critiques, in a loop, when you have clear evaluation criteria.

Notice that in the first four the code decides what happens next, and in the last two the model has more freedom. You can mix them.

Where prompted control flow breaks (my experience)

A prose-encoded state machine degrades gracefully for a human reader and badly for a model under load. A long system prompt with a dozen conditional branches reads fine when you proofread it once. In production, small context shifts (a slightly different tool result, an extra retry, a user message that answers two steps at once) can make the model skip a step, redo one, or invent a transition you never specified. None of it raises an error. You get an agent that quietly does the wrong thing and sounds confident about it.

The tell is in your logs: agents that "usually" follow the happy path but occasionally loop, skip a required check, or act out of order, with no single bad input to blame.

What to move into code

Any transition that is deterministic given the current state belongs in code. "After payment succeeds, always send the confirmation before offering upsells" is a rule, not a judgment, so write it as an explicit state machine. Keep the model for the parts that need judgment: interpreting ambiguous input, choosing between valid next actions, writing the user-facing text. The state machine decides what is allowed next, and the model decides how to do the allowed thing.

That split also makes testing far cheaper. A state machine's transitions can be unit-tested with no model in the loop: assert once that an order in awaiting_payment can move only to paid or cancelled, instead of hoping every prompt revision keeps that invariant. For the agent's remaining choices, see Trajectory Tests for Agents.

A rule of thumb

If you can draw your agent's behavior as a flowchart with a small number of named states and legal transitions, don't leave its enforcement to a paragraph of instructions. Write it as code. Let the prompt do what prompts are good at, reasoning within a step, not remembering which step comes next.

Sources

AI AgentsAgentic AISoftware Engineering