Canary Queries for LLM Features: Cheap Always-On Probes That Catch Quality Drift

Most LLM features are monitored the way ordinary web services are: error rates, latency, maybe token spend. Those signals tell you the service is *up*. They say very little about whether the feature is still *good*. A canary query closes part of that gap: a small, fixed set of requests with known-good expectations, sent to the real production path on a schedule, so you notice quality drift before your users report it.

What a canary query is (and isn't)

A canary is not an eval suite. An eval suite is broad, expensive, and run before a change ships. A canary is narrow, cheap, and run continuously against what is *already* shipped. It answers one question: "Does the live path still do the thing it did yesterday?"

Good canaries share a few properties:

• Deterministic expectations. Pick requests where correctness can be checked mechanically: the output parses against the schema, contains a required field, cites a document that must be retrieved, calls the expected tool, or stays under a length limit. Avoid canaries that need another model to grade them; you would be monitoring two moving parts.
• Coverage of the seams. Include one canary per integration point that can break independently: the retrieval step, each tool, the output parser, the provider fallback path. A single "hello world" prompt misses most failures.
• Realistic shape. Use inputs that resemble real traffic in length and structure, not toy strings. Many regressions only appear with long contexts or messy formatting.

Wiring it up

Run the canaries through the same code path production uses, including the same prompt template, model identifier, and middleware. If the canary calls the API directly while production goes through a wrapper, you are testing something your users never touch.

Tag canary traffic so it is excluded from product analytics and billing dashboards, but *not* from your tracing. You want to be able to pull up a failed canary's full trace the same way you would for a real request.

Record three things per run: pass/fail per assertion, latency, and the raw output. The raw output matters because a canary that starts failing often fails in a way you can diagnose by eye in ten seconds.

Set alerting on *trends and repeats*, not single failures. Models are nondeterministic, so one failed canary out of twenty is noise; the same canary failing three runs in a row, or a pass rate dropping below a threshold you chose in advance, is a signal. Deciding that threshold up front keeps the alert from being argued down at 2 a.m.

What canaries catch

In practice, canaries tend to surface a recognizable set of problems: a provider-side behavior change behind an unchanged model alias, a prompt template edit that broke a rarely exercised branch, a retrieval index that quietly stopped refreshing, an expired credential on a secondary tool, or a parser that no longer handles a new output habit.

They will not catch subtle quality erosion, bias, or problems that only appear on inputs you did not think to include. That is fine. Canaries are a tripwire, not a judge.

The closing rule

Keep the set small enough that you actually read the failures, and review it whenever you add a new integration point. A canary suite nobody looks at is just another source of alerts people learn to ignore. Ten well-chosen canaries that someone trusts will protect you more than two hundred that nobody does.