A prompt change moves your eval pass rate from 86% to 89%. You ship it. A week later, support reports that long documents are being mangled, and nobody can see why: the eval said things got better. They did, on average. The average was dominated by short, easy inputs, and the hard slice got worse.

An aggregate score answers "is the system better overall?" Almost nobody actually needs that question answered. You need to know *for whom* it got better and *for whom* it got worse.

Tag cases when you write them

Slicing is only possible if each eval case carries metadata. Add a small, flat set of tags at the moment a case is created, because reconstructing them later is tedious and inconsistent. Useful dimensions for most LLM features:

  • Input size: short, medium, near-the-context-limit.
  • Input source: clean typed text, pasted text with formatting debris, OCR output, translated content.
  • Task variant: if one prompt handles several intents, tag the intent.
  • Difficulty or ambiguity: cases where reasonable humans disagree versus cases with one clear answer.
  • Origin: hand-written, sampled from production, or added after a bug. Bug-derived cases are your regression guard and deserve their own row in the report.

Keep the vocabulary small and fixed. Ten tags used consistently beat fifty used loosely, and a free-text tag field drifts into synonyms within a month.

Report the table, not the number

Have the runner emit a grid: one row per slice, columns for count, pass rate, and the delta against the baseline run. A comparison might look like this:

| Slice | n | Baseline | Candidate | |---|---|---|---| | short input | 120 | 94% | 96% | | long input | 30 | 70% | 57% | | bug-derived | 18 | 100% | 94% |

The overall number improved, yet the long-input slice lost thirteen points. Without the table, that change would have shipped.

Two cautions. First, slices are small, so their pass rates are noisy. A 13-point drop on 30 cases is a prompt to look at the failing cases, not a proof. Treat the slice table as a way to decide *where to read transcripts*, and read them. Second, slicing many ways multiplies the chance that something looks bad by luck alone, so resist alarm on a single slice that moved a little, and pay attention to slices that move together or that match something you changed on purpose.

Let slices drive what you add

When a slice is thin, you have found a gap in the eval set, not just the product. If "pasted text with formatting debris" has six cases and accounts for a third of production traffic, the eval is under-measuring the thing your users do most. Sample real examples into that slice until the case count roughly tracks how much it matters.

The closing rule is simple: no change gets declared an improvement from the overall number alone. Require the slice table in the review, and require that no slice you care about regressed without an explicit, written decision to accept it.

Large Language ModelsSoftware Engineering