Ordered target statistics encode a categorical value from target values observed earlier in a randomly permuted sequence. Because the current example is excluded from its own history, its target cannot directly influence its encoding. section 8.2.1
Compute the encoding procedurally:
- Randomly permute the training examples into an order. section 8.2.1
- Treat the examples as arriving sequentially and use only the preceding examples as the available history. section 8.2.1
- For each example, calculate the category’s target statistic from that preceding history, never using the example’s own target. section 8.2.1
A small permutation
Suppose the randomized order is 4, 7, 1, 8, 2, 6, 5, 3. Encode example 2 using examples 4, 7, 1, and 8; encode example 6 using those examples plus 2. Early examples have much less history, so their encodings are based on fewer observations. The approach resembles leave-one-out encoding but uses a sequential history rather than all other examples. section 8.2.1 section 8.1.3
Greedy target encoding can leak training targets and contribute to distribution shift. Hold-out encoding avoids these problems but wastes data; ordered target statistics provide a data-efficient alternative that further mitigates leakage and prediction shift. CatBoost combines this encoding with ordered boosting. section 8.5 section 8.3.4
Figure 8.8 Ordered target statistics first permutes the examples into a random sequence, using only the previous examples in the ordered sequence to compute the target statistics.
Use the permutation diagram to follow how each example receives statistics from only the earlier examples in the randomized sequence. section 8.2.1
