A team tells you their AI feature "catches 95% of errors." It sounds like a result. It is really a sentence with a missing noun, and before you act on it you should ask: 95% of what?
Of errors in a curated test set of 40 examples? Of errors users reported, which excludes every error nobody noticed? Of errors the system flagged as uncertain, which excludes the confident mistakes? Each reading gives the same headline and a very different product.
Three habits make metrics honest.
State the denominator beside the number. "38 of 40 curated cases" is less impressive than "95%", and also much more useful. It tells the reader the sample is small, which tells them how much to trust a one-point change.
Say what was excluded, and why. Every evaluation filters something: inputs that failed to parse, requests that timed out, conversations that ended early. Those exclusions are often where the worst behavior lives. If a metric only covers the cases that completed successfully, say so, and report how many did not.
Track the count of "could not evaluate." A rising number of unscorable outputs can make a success rate climb while quality falls, because the hard cases quietly drop out of the calculation.
None of this requires sophisticated statistics. It requires writing two numbers where you used to write one, and being willing to let the second one look unimpressive.
The next time a dashboard shows a percentage, pick one and ask for its denominator. If nobody can state it in a sentence, you have found the first thing to fix.