liveBook · Manning

Benchmark contamination

As explained in Reinforcement Learning from Human Feedback

Dataset contamination is the inclusion of evaluation examples or closely related material in a model's training data, invalidating independent evaluation.

Benchmark contamination occurs when evaluation examples or closely related material enter training data, turning benchmark scores into confounded measures rather than independent evidence of model capability. It commonly arises through web scraping, user-provided prompts, or repeated reuse of public evaluations across training stages. section 16.4 section 16.0

Detect and manage contamination through practical checks:

  • Search training and evaluation data for n-gram or subword-token overlap, or fixed-length character matches such as 50-character substrings. section 16.4
  • Treat close variants as potential contamination: preserving a mathematics problem while changing only its numbers can still produce misleading benchmark behavior. section 16.4
  • Interpret contaminated or undisclosed training histories cautiously, because anomalous gains may reflect memorized benchmark structure rather than the intended training signal. section 16.4 section 16.6

Tülu 3 decontamination findings

An 8-gram comparison between training prompts and evaluation prompts identified overlaps between UltraFeedback and TruthfulQA, Evol-CodeAlpaca and HumanEval, NuminaMath and MATH, and WildChat and safety evaluations. Such matches can invalidate apparently independent results. section 16.4

When training data cannot be verified, newly generated benchmarks with slightly perturbed questions provide a way to probe memorization, although score comparisons remain approximate because prompting, sampling, and evaluation formats also affect results. section 16.4 section 16.6

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Defines contamination, gives overlap-matching procedures, and examines near-duplicate effects, but does not establish universal leakage-proof evaluation.
Examples
Uses Tülu 3 overlap findings, altered-number math problems, and random-reward reinforcement learning as concrete contamination probes.
What it assumes
Assumes familiarity with language-model training and benchmarks while defining contamination and decontamination directly.
What you can do after
Diagnose/debug with it by matching training and evaluation text, flagging near-duplicates, and qualifying scores from potentially contaminated models.
How it connects
Connects dataset contamination with data leakage, decontamination, web scraping, RLHF, post-training, and benchmark evaluation.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.