Data leakage occurs when evaluation or otherwise unavailable information enters training. The model can then appear accurate by memorizing answers it should never have seen, producing overfitting and poor generalization. section 5.3.3
Leakage can arise during continual RLHF: evaluation prompts may later be selected for human feedback and reinforcement training, or users may ask questions resembling evaluation prompts. Unrestricted updates turn evaluation into a route for training contamination. section 5.3.3
Prevent leakage with practical controls:
- Define train and validation splits before training, then shuffle carefully so examples cannot cross split boundaries. section 9.1.2
- Treat evaluation data as unavailable training information, especially when periodic production evaluations feed later human feedback. section 5.3.3
- Prefer controlled, periodic model updates when continual RLHF could invalidate prompting or retrieval-augmented generation work. section 5.3.3
A misleading evaluation
Suppose an evaluation prompt is later sampled for human feedback and used in RL training. The model may answer that prompt well because it encountered the test case during training—not because it generalizes. This resembles hiring programmers who recognize familiar LeetCode problems without demonstrating job-relevant ability. section 5.3.3
Listing 9.2 Loading and preparing the data
dataset = load_dataset( #1
"text",
data_files={
"train": ["../../data/TinyStoriesv1andv2-train.txt"],
"val": ["../../data/TinyStoriesv1andv2-valid.txt"],
},
streaming=True,
)
The train and validation file declarations make the intended split boundary explicit before streaming and preparing the dataset. section 9.1.2
