Target statistics encode a categorical feature with a numerical summary of its associated labels, usually their mean. Because the encoding uses targets, it is supervised. Unlike one-hot encoding, it preserves the number of columns; unlike ordinal encoding, it does not impose artificial category order. section 8.1.3 section 8.1
Greedy target encoding
Suppose red T-shirts have costs 8.99, 9.99, and 25.00. Their target statistic is the mean, 14.66, so every red value becomes 14.66. Blue and green are replaced by their own category means. section 8.1.3
Greedy encoding uses all training labels, which can leak target information and create distribution or prediction shift when the resulting encoding is applied to test data. Hold-out encoding reserves a separate encoding set; leave-one-out encoding excludes only the example currently being encoded. section 8.1.3 section 8.5
Ordered target statistics reduce these problems procedurally:
- Randomly permute the training examples. section 8.2.1
- For each example, compute its category statistic using only preceding examples in the permutation. section 8.2.1
- Use CatBoost to combine ordered target statistics with categorical combinations and ordered boosting. section 8.2.1 section 8.5
