Train-test leakage occurs when target information from training-related data enters the encoding or evaluation process for test examples. The model then appears to generalize better than it will on future unseen data. section 8.0 section 8.5
Target encoding makes the risk especially clear:
- Greedy target statistics replace each category with a statistic such as its target mean, using all training data for encoding; this creates leakage and distribution shift. section 8.5
- Hold-out target statistics reserve a separate encoding set in addition to the test set, eliminating leakage and shift but wasting data. section 8.5
- Leave-one-out and ordered target statistics use more data-efficient procedures to mitigate leakage and shift. section 8.5
- CatBoost implements ordered target statistics and ordered boosting, which use permutations and subsets to reduce leakage-related prediction shift. section 8.0 section 8.5
Why evaluation becomes misleading
Suppose a category is encoded using target information from all training rows before evaluation. Because the encoding reflects outcomes unavailable when predicting genuinely unseen examples, the measured generalization performance can be overly optimistic. Separating encoding data, leaving observations out, or ordering the statistics prevents that information from flowing into the evaluation. section 8.5
