Most LLM features see a mix of easy and hard requests. A cascade exploits that: try a small, cheap model first, and escalate to a larger one only when the first answer looks unreliable. Done well, the bulk of traffic never touches the expensive model. Done badly, you ship a worse product and your dashboards say everything is fine.

The whole design is the escalation signal

A cascade is only as good as its answer to one question: *how do we know the cheap answer is not good enough?* There are a few honest options, roughly in order of reliability:

  • Deterministic validation. Does the output parse, match the schema, pass a unit test, or cite a source that actually exists in the retrieved set? Failures escalate. This is the strongest signal because it does not depend on a model grading itself.
  • Agreement checks. Ask the small model twice, or ask two cheap models, and escalate on disagreement. This costs more per request but catches unstable answers.
  • Self-reported confidence. Cheap to implement and the least trustworthy. Models are often fluent and wrong in the same breath. If you use it, treat it as a weak hint and calibrate the threshold against labeled data rather than picking 0.8 because it feels right.
  • Request features. Input length, number of entities, or a keyword that marks a known hard category. Crude, but stable and easy to reason about.

Prefer signals you can verify mechanically, and combine them. A request that fails validation *or* trips a request-feature rule escalates.

Measure the cascade, not the models

Evaluating each model alone tells you little. What matters is the end-to-end behavior of the routed system:

  1. Quality at the final output. Run your eval set through the full cascade and compare against always using the large model. Report the gap, sliced by request type, because an average can hide a category that the small model quietly fails.
  2. Escalation rate. If 90% of traffic escalates, you pay for two calls and gain nothing. Track the rate over time; a sudden shift usually means the input distribution changed.
  3. Cost and latency per successful task. An escalated request pays for both calls and waits for both. Check that the tail latency is acceptable, not just the mean.
  4. Missed escalations. This is the dangerous one. Sample requests the cascade *accepted* from the small model and have the large model or a human review them. Without this audit, you only ever see the errors the system already knew about.

Practical guardrails

Keep the router's decision in your logs next to the request, so you can reconstruct why a given answer came from the small model. Make the threshold a config value, not a constant, so you can tighten it without a deploy. And re-run the evaluation whenever either model version changes, since the escalation signal was calibrated against a specific pair.

The takeaway: a cascade is a bet that your cheap-answer check is better than chance. Build the check first, measure the missed escalations, and only then trust the savings.

AI IntegrationSoftware Engineering