Swapping a model or rewriting a major prompt is one of the riskiest changes in an AI feature, and offline evals only cover the cases you thought to write down. Shadow mode closes part of that gap: the new configuration runs against real production traffic, its outputs are recorded, and users never see them.

What shadow mode is, and what it is not

For each live request, the system serves the answer from the current ("primary") configuration as usual. In parallel or just afterward, it sends the same input to the candidate and stores the result alongside the primary's. Nothing from the candidate reaches the user.

This is different from an A/B test. An A/B test measures user outcomes and exposes some users to the candidate. Shadow mode is a safety and quality screen that comes first: it tells you whether the candidate is worth exposing anyone to.

Making it safe to run

Shadowing adds load and risk, so build in guardrails:

  • Never block the user path. Run the shadow call asynchronously with its own timeout, and make sure its failure cannot fail the request.
  • Watch side effects. If the candidate can call tools, give it read-only or stubbed versions. A shadow agent that sends a real email is not a shadow.
  • Sample. Ten percent of traffic is often enough, and it caps cost. Sample by request ID so the same sampled set is comparable over time.
  • Respect data rules. Stored shadow outputs contain user content. They deserve the same retention and access controls as your primary logs.

What to compare

Raw side-by-side diffs of free text are hard to read at scale. Prefer comparisons that reduce to numbers:

  1. Hard checks: schema validity, refusal rate, truncation rate, tool-call error rate, length distribution, latency, and token cost.
  2. Agreement: for classification or extraction, how often do the two configurations agree? Disagreements are the interesting set.
  3. Judged samples: take a sample of disagreements and review them, with humans or a calibrated LLM judge, to decide which side was right.

The disagreement set is the real product of shadow mode. A high agreement rate with a few well-understood differences is a strong signal. A low rate is not automatically bad, but it means you must read the cases before proceeding.

Turn findings into regression tests

Every disagreement where the candidate was worse should become an eval case, with a note on why. That way the next candidate is tested against what this one taught you, and the shadow run leaves behind a permanent improvement rather than a one-time report.

Decide the exit criteria in advance

Before starting, write down what "ready" means: for example, no increase in invalid outputs, cost within an agreed band, and no category of disagreement the team judges as a regression. Without criteria agreed up front, a shadow run tends to end with a debate over whichever numbers look best.

The takeaway

Shadow mode converts "we think the new model is better" into evidence gathered on your real traffic, at no risk to users. It costs extra compute and some engineering care, and for a change that touches every request, that is usually a bargain.

Large Language ModelsSoftware Engineering