Eval sets rarely fail dramatically. They decay. Someone adds forty cases during a bad week, someone else adds a batch scraped from logs, and a year later the set has two hundred entries, a pass rate that wobbles, and nobody who can say which failures matter. The root cause is usually mundane: each case is just an input and an expected output, with no record of why it exists.

Give every case a reason

Add a small amount of metadata to each case and you change what the set can do for you. Four fields cover most of the value:

  • Why it exists. One sentence: "Regression: model invented a refund policy when none was in context," or "Boundary: input at the maximum supported length." If you cannot write the sentence, the case probably should not be there.
  • Origin. Hand-written, sampled from production, or synthetic. This matters for contamination and for how much you should trust a pass.
  • Severity. What happens in the product if this fails: cosmetic, wrong answer, or harmful. A flat pass rate treats all of these as equal, which they are not.
  • Expected stability. Is this case deterministic enough to gate a release, or is it a noisy probe you only want to watch as a trend?

What this buys you

Triage becomes fast. When a run goes red, you can group failures by reason and severity instead of reading raw diffs. Three failures in the "invented policy" group tell a very different story than three scattered ones.

Deletion becomes safe. Pruning is the hardest part of maintaining an eval set because removing a case feels like lowering the bar. With a recorded reason, you can retire a case when the condition that motivated it is gone, such as a feature that was removed, and note that in the history rather than deleting silently.

Coverage gaps become visible. Counting cases by reason reveals that, say, you have thirty tests for formatting and two for refusals. That is a conversation worth having, and you can only have it if the categories are written down.

Reviews improve. A pull request that adds cases is now reviewable. The reviewer asks whether the reason is real and whether the expected output actually follows from it, rather than rubber-stamping a data file.

Keep it light

This does not need a platform. A JSONL file with id, input, expected, reason, origin, severity and gate fields is enough, plus a tiny script that fails CI if a new case lacks a reason. Report results sliced by severity and by reason so the headline number never hides the one group that matters.

The closing point: an eval set is documentation of what you promised your users. Cases without reasons are promises nobody remembers making, and those are the ones that get broken quietly.

Large Language ModelsSoftware Engineering