Why Your AI Feature Needs a Failure Taxonomy, Not Just an Error Rate
Most teams track one number for their AI feature's health: an error rate, or a success rate on evals. That number tells you something is wrong long before it tells you what. And "what" is the part that actually determines whether Monday's fix is a prompt tweak, a retrieval change, a schema fix, or a decision to route around a flaky upstream dependency entirely.
A failure taxonomy is just a short, explicit list of the distinct ways your feature fails, applied consistently when you review real outputs: the model called the wrong tool, the model called the right tool with bad arguments, the retrieved context was irrelevant, the retrieved context was relevant but the model ignored it, the output violated your schema, the output was correct but too slow, the user's request was genuinely out of scope. Five or six categories is usually enough to start.
The value shows up the moment you have more than a handful of logged failures. Without a taxonomy, "we're seeing more errors this week" triggers a scramble across the whole stack. With one, "tool-argument errors are up 40%, everything else is flat" points a team at one specific fix, on day one, instead of a general re-investigation.
It also changes what you prioritize. A feature with a 90% success rate where every failure is the same well-understood cause is in much better shape than one with a 95% success rate spread across a dozen distinct, poorly-understood failure modes — and an aggregate error rate can't tell those two situations apart.
Start small: the next time you review a batch of failures, don't just count them. Name them.