Most teams build a prompt and an eval set in the same week, from the same pile of real examples. That is convenient, and it quietly corrupts the measurement. If an example appears in the prompt as a demonstration and again as a test case, the model is not being tested on the task. It is being tested on whether it can copy an answer it was just shown.

How the leak happens

The obvious version is an exact duplicate: someone pastes a failing case into the prompt as a fix, and the case stays in the test suite. The subtler versions are more common:

  • Near-duplicates. The prompt example and the test case differ by one name or number but share the same structure and the same answer.
  • Shared source documents. Both come from the same customer ticket, contract, or codebase, so the surface vocabulary overlaps heavily.
  • Fix-by-example drift. Every regression you patch by adding an example makes the suite look better without making the feature more general.

The symptom is a score that rises steadily while users keep reporting new failures. The eval says 94%, production feels like 80%, and nobody can explain the gap.

A cheap process that prevents it

You do not need a research-grade decontamination pipeline. Three habits cover most of the risk.

  1. Split before you write the prompt. Divide your collected examples into a development pool and a held-out pool first. Only the development pool may ever be copied into a prompt. Record which case IDs live where.
  2. Check mechanically. Add a test that fails the build if any held-out case ID, or any case whose normalized text has a high overlap with a prompt example, appears in the prompt file. Even a simple normalized-string and n-gram comparison catches the paste-it-in-as-a-fix mistake.
  3. Refresh the held-out pool. A held-out set that has been looked at for months stops being held out, because every engineer has read the failures and unconsciously shaped the prompt around them. Periodically replace a slice with fresh production samples, and keep the older slice as a regression set that you label as such.

When you do fix a failure by adding an example, write a *new* test case that probes the same failure from a different angle, rather than reusing the case that motivated the fix. The motivating case is now training data. The sibling case is the test.

The closing point

Treat the boundary between "what the prompt has seen" and "what the eval measures" as a real interface, with an owner and an automated check, the same way you treat the boundary between training and validation data in classical machine learning. An eval that can be improved by editing the prompt toward its own test cases is not an eval. It is a second copy of the prompt.

Large Language ModelsSoftware Engineering