A held-out test set is the one number you are supposed to be able to trust. You never tune against it, the model never sees it, and so its score is an honest estimate of how the model behaves on new problems. That promise is easy to make and surprisingly hard to keep, for the labs that build models and for the teams that evaluate them.

What "contamination" means

Researchers call the failure benchmark data contamination: evaluation data leaking into the training data. The surveys on the topic describe it as an undesirable phenomenon that inflates a model's measured performance on the evaluation data. It comes in two flavours. Input contamination means the test questions appeared in training; input-plus-label contamination means the questions and their answers did. The second is the more damaging one, because the model can simply have memorized the answer.

The usual cause is unglamorous. Pretraining data is collected by scraping the web at enormous scale, with minimal oversight of what any single page contains. Once a benchmark is popular, it is discussed: a forum post works through a problem, a blog explains the solution, a paper quotes a question in an appendix, a dataset gets mirrored to a code-hosting site. Nobody has to intend to cheat for those pages to end up in the next training run.

Why you can't just check

For a model whose training data is not public, confirming contamination directly is not possible, because you would need the training corpus. The research literature splits detection methods into two broad families, matching-based and comparison-based, and separates open-data settings (you can see the corpus) from closed-data settings (you can't). Closed-data detection is necessarily indirect.

The practical, indirect signals are the ones you can run yourself:

  • Paraphrase test. Rewrite benchmark questions so they test the same skill in different words. A model that does well on the original wording and drops sharply on the paraphrase is leaning on memorization.
  • Fresh-problem test. Compare performance on problems that plausibly did not exist when the model's training data was collected against older problems. A large gap is a warning.
  • Verbatim completion. Give the model the first half of a benchmark item and see whether it reproduces the rest word for word.

None of these proves contamination on its own. They are evidence you weigh together.

The same problem hits your own eval set

Most teams reading this are not training models; they are evaluating prompts, retrieval settings, or model choices against a test set they built. The same logic applies, and the leak is usually self-inflicted:

  • You tuned on the test set. Every time you look at failures on the "held-out" set and adjust the prompt, the set becomes a validation set, and its score becomes optimistic. Keep a separate split you do not look at until the end.
  • Examples drift into the prompt. A failing test case gets copied into the prompt as a few-shot example "to fix it," and now the test measures memory, not skill.
  • The set is too small and too old. A few dozen cases that never change get overfit by iteration alone, even with no literal copying.

What to do about it

My recommendations, drawn from the contamination literature's general logic rather than any single study:

  1. Treat a public leaderboard number as a hypothesis, not a verdict. It is a useful starting point for shortlisting models, not a prediction of how one will behave on your data.
  2. Build a private set from your own traffic. Real inputs from your application, labelled by your team and never published, are the strongest defence, because they can't have been in anyone's training data.
  3. Split three ways. Keep a development set you iterate on, a validation set you check occasionally, and a final test set you touch rarely.
  4. Refresh it. Add new cases regularly so a prompt can't be overfit to a frozen list, and retire cases that every model now passes.
  5. Add paraphrase variants of important cases. If scores swing with wording, you have learned something real about robustness.

The takeaway

Whether you are reading a model card or running your own evaluation, the right question is not "what score did it get" but "could it have seen this before, and how would I know." A score you can't defend against that question is a lead worth following up, not a result to build a decision on.

Sources