Data leakage occurs when evaluation information, especially test or validation data, enters training or model-selection decisions. The resulting performance estimate is optimistically biased and may fail in production. section 7.4.1
Control leakage through explicit modeling and testing work:
- Keep test information out of training and avoid repeatedly optimizing against validation quirks. section 7.4.1
- Inspect training and validation choices for time travel, especially when future observations can signal outcomes in evaluation data. section 7.4.1
- Create a testing ticket that defines test-data creation and the controls used to prevent leaks in a reproducible test environment. section 8.2
- Treat unexpectedly strong results as a model smell: check pipeline bugs, toolkit issues, configuration mistakes, and overfitting before productionization. section 7.5
A hot-summer validation split
A smart-building model uses data from a hot summer for validation, while the test set comes from the same summer. The model appears accurate because the split shares a seasonal signal that will not recur in production. section 7.4.1
Structured experiments, records, and automated testing make leakage easier to detect and enable fast reruns after pipeline or configuration defects are fixed. Multi-armed-bandit testing offers an alternative that is described as resistant to information leakage, though it does not quantify how much better one model is. section 7.5 section 8.2.6
Figure 8.2 Three one-arm bandits with pay-out records, where each block represents one pull. A light record indicates a zero reward, and a dark one indicates a positive reward. Machine A pays frequently, Machine B pays occasionally, and Machine C never pays.
Compare competing machines by observed reward records: the visual supports the bandit-based alternative to repeatedly optimizing against a fixed evaluation set. section 8.2.6
