A prompt change goes from 72% to 75% on your eval set, and the pull request gets approved. The set has 50 examples. That three-point gain is one and a half questions. Before you treat it as progress, ask how much the score would move if you ran the same system on a different 50 questions from the same distribution.
Rough intuition for the size of the noise. For a pass/fail metric near 70%, the standard error on 50 examples is around 6 percentage points, so an honest interval around any single score spans roughly plus or minus 12 points. On 500 examples it shrinks to about plus or minus 4. On small sets, most of the differences we celebrate sit inside the noise.
Paired comparisons help, but don't erase the problem. Because both versions run on the same questions, you can look only at the examples where they disagree. If the new prompt fixes 5 and breaks 3, that is 8 disagreements, and a 5-to-3 split is what you'd see from a coin flip most of the time. A 12-to-2 split is a signal.
Model nondeterminism adds a second source. If a single question can flip between runs at nonzero temperature, run each example more than once, or report how often results flip before you interpret a difference.
What to do about it, in order of cost:
- Report counts alongside percentages: "fixed 5, broke 3" is more honest than "+3 points."
- Set a decision rule before looking, such as "ship only if net fixes exceed breaks by a margin, and no critical category regresses."
- Grow the set where decisions are close. New production failures are the cheapest source of additional examples.
- Read the disagreements. Even a noisy aggregate can point at a real pattern once you see what actually changed.
None of this means small evals are useless. They're good at catching large regressions and at surfacing individual failures to read. They're poor at ranking two similar candidates. Use them for the first job, and be willing to say "we can't tell yet" for the second.