Target encoding replaces a category with a statistic computed from its target values. A hold-out encoding set keeps those target values separate: it is reserved exclusively for computing encodings applied to the training and test sets. section 8.1.3
Use the partitioning procedure to keep target statistics independent:
- Split the available data into training, hold-out encoding, and hold-out test sets. section 8.1.3
- Compute category target statistics using only the hold-out encoding set. section 8.1.3
- Apply the same statistics to the training and test sets, preserving independence and avoiding prediction shift caused by different encoding sources. section 8.1.3
- Discard the encoding set after computing the statistics, making the method potentially wasteful of data. section 8.1.3
A three-way color example
Suppose a categorical feature contains the value red. Instead of using training labels greedily to encode red, reserve a third subset for computing its target statistic. Use that statistic for both training and test examples with red. The arrangement prevents the training labels from leaking into the encoding and keeps the encoded distributions aligned.
Leave-one-out encoding reduces data waste by excluding only the example currently being encoded, but it does not completely mitigate prediction shift. CatBoost instead uses permutation-based ordered target statistics to avoid relying on a permanently discarded subset. section 8.1.3 section 8.2.1
