Every eval number is a ratio: a score divided by a set of cases. We tend to track the score carefully and the set barely at all. That is how a team ends up celebrating a four-point improvement that came from deleting the hard cases.
The set changes more than you think
Eval sets are living files. Someone adds examples from a recent incident. Someone removes a case because the label looked wrong. Someone fixes a typo in an input, or re-labels a borderline example. Each edit is reasonable on its own. Together they mean that last month's 82% and this month's 85% were measured against different exams.
If you only store the score, you cannot tell a real improvement from a changed denominator. And because the change usually happens quietly, nobody remembers to ask.
A lightweight fix
Keep a changelog for the eval set, just as you would for an API. It does not need tooling to start:
- Give the set a version, and record that version next to every reported score.
- Log each change with a date, a reason, and the case IDs affected: added, removed, edited, relabeled.
- When you make a change that could move the score, re-run the previous version of the set with the current system, so you have a like-for-like baseline.
- Never delete a failing case without writing down why. "The label was wrong" is a legitimate reason; "the model could not do it" is not.
Cases added from real incidents deserve special care. They are valuable because they fail, so your score will probably drop when you add them. Record that drop as a feature of the new version, not as a regression in the system.
The payoff
When someone asks why the number moved, you can answer in a minute: the system changed, the set changed, or both, and here is the evidence. That answer is what makes a metric trustworthy enough to base decisions on. A score without a set version is just a number; with one, it is a measurement you can compare.