Supervised encoding

As explained in Ensemble Methods for Machine Learning

Supervised encoding is a categorical-feature encoding technique that uses target values when deriving category representations.

Supervised encoding represents each category using both feature values and target labels. Unlike unsupervised encodings, it can capture how categories relate to the prediction target. section 8.1 section 8.1.3

A basic target-statistics procedure works as follows:

• Choose a categorical feature and a statistic, such as the mean target value. section 8.1.3
• Group training examples by category and compute the statistic for each group. section 8.1.3
• Replace every category with its corresponding numerical statistic; unlike one-hot encoding, this adds no columns. section 8.1.3
• Prevent leakage with hold-out, leave-one-out, or ordered target statistics rather than using every label indiscriminately. section 8.5

Encoding T-shirt color

For red T-shirts costing 8.99, 9.99, and 25.00, the mean target is 14.66. Every red value is therefore replaced by 14.66; blue and green receive their own category means. section 8.1.3

Figure 8.2 The category red of the feature color is replaced by its target statistic, the average (mean) of all the target values (cost) corresponding to the examples whose color is red. This is called greedy target encoding as all the training labels have been used for encoding.

Visualizes greedy target encoding by replacing red with the mean cost of all red examples. section 8.1.3

Figure 8.5 Hold-out encoding partitions the available data into three sets: training and test, as usual, and a third hold-out test set to be used exclusively for encoding with target statistics. This avoids both target leakage and distribution shift.

Shows how a separate encoding hold-out set avoids target leakage and distribution shift. section 8.1.3

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains category-wise target statistics, greedy leakage, and hold-out or leave-one-out safeguards.

Examples

Demonstrates encoding red, blue, and green T-shirt colors using their mean costs.

What it assumes

Assumes categorical features, labels, and basic encoding methods such as one-hot and ordinal encoding.

What you can do after

Apply target statistics while recognizing and mitigating leakage during evaluation.

How it connects

Contrasts with one-hot encoding and ordinal encoding, and connects to target leakage, distribution shift, and ordered boosting.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• mean target encodingincludes · Mean target encoding is a categorical encoding technique that replaces each category with the mean of its associated target values.
• target statisticsuses · Target statistics encode a categorical feature with a statistic computed from target values associated with each category.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.