Mean target encoding

As explained in Ensemble Methods for Machine Learning

Greedy target encoding uses target-derived statistics computed from training data to encode categorical values, potentially introducing target leakage when the encoded examples contribute to those statistics.

Mean target encoding replaces each categorical value with the average target observed for that value in training data. For a cost prediction task, every red T-shirt can become the mean cost of the red T-shirts. It is supervised encoding: labels, rather than category names alone, determine the numeric representation. section 8.1.3

This offers a compact alternative to one-hot encoding. It keeps the original number of feature columns instead of adding one column per category, and it avoids the arbitrary ordering that ordinal encoding imposes on otherwise unordered categories. The apparent simplicity has an important catch: the target values used to make the encoding can reveal information about examples being encoded. section 8.1.3 section 8.5

Encoding T-shirt color

Suppose red T-shirts have costs 8.99, 9.99, and 25.00. Their mean is 14.66, so each red value is replaced with 14.66. Applying the same calculation to blue and green yields their own category means, 16.82 and 13.99. This use of all available training labels is greedy target encoding. section 8.1.3

A greedy encoding workflow is straightforward:

• Split features from the target and identify categorical columns. section 8.1.4
• For each category, calculate its target mean from the training data. section 8.1.3
• Replace category values with their corresponding means while preserving the feature count. section 8.1.3
• Fit the downstream model on encoded training features and evaluate it on encoded test features. section 8.1.4

Greedy encoding is risky when a training example helps compute the mean later assigned to itself: its own label has entered its feature value. Using training-derived encodings for test data can also create a distribution shift between encoded training and test sets. Both effects can distort estimates of generalization performance. section 8.1.3 section 8.5

Figure 8.2 The category red of the feature color is replaced by its target statistic, the average (mean) of all the target values (cost) corresponding to the examples whose color is red. This is called greedy target encoding as all the training labels have been used for encoding.

Inspect the red-category calculation to see exactly how labels become a single numeric feature value. section 8.1.3

Safer target-statistic approaches separate the data used to construct an encoded training value from that example’s own target. Hold-out encoding reserves a third split for encoding, avoiding leakage and shift but using less data. Leave-one-out encoding excludes only the example currently being encoded; ordered target statistics are another data-efficient mitigation. section 8.5 section 8.1.3

In a scikit-learn-compatible pipeline, category_encoders provides TargetEncoder for greedy target encoding and LeaveOneOutEncoder for its leave-one-out variant. On Australian Credit Approval data, greedy target encoding reduced the post-encoding feature count from 38 under one-hot encoding to 14, while a random forest example reported 91.304% accuracy versus 89.855% for one-hot encoding. Such a result is a comparison to evaluate with appropriate metrics, not a guarantee that greedy encoding generalizes best. section 8.1.4 section 8.1.4 section 8.3.3

Encoder choice interacts with the model and metric. In comparisons across random forest, LightGBM, and XGBoost, target encoding was particularly effective by balanced accuracy and F1 with random forest, while LightGBM results were roughly similar across one-hot, ordinal, and target encodings. Compare validation metrics that account for class imbalance rather than relying only on plain accuracy. section 8.3.3

How Ensemble Methods for Machine Learning covers it

How deep it goes

Defines per-category target means, walks through a numeric example, and contrasts leakage mitigations without deriving their implementations.

Examples

Uses T-shirt color and cost means, Australian Credit Approval pipelines, and encoder comparisons with random forest, LightGBM, and XGBoost.

What it assumes

Assumes labeled categorical data and train-test evaluation while reinforcing categorical encoding and class-imbalance-aware metrics.

What you can do after

Apply it via category_encoders and evaluate it against alternative encoders with classification metrics.

How it connects

Connects target encoding to one-hot encoding, ordinal encoding, target leakage, distribution shift, leave-one-out encoding, and ordered target statistics.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA feature encoding technique that replaces each category with a statistic computed from the target values associated with that category.
• 

  liveBookmanning.com
  Machine Learning System DesignTarget encoding is a categorical-variable encoding technique that replaces each category with a statistic of the target values associated with that category.

Compare how 3 books explain mean target encoding →

Related concepts

• TargetEncoderimplemented by · TargetEncoder is an encoding method that replaces categorical values with statistics computed from the target variable, often using smoothing.
• target leakingproduces · Target leakage is the inclusion or use of information derived from the prediction target that would be unavailable when making predictions.
• supervised encodingkind of · Supervised encoding is a categorical-feature encoding technique that uses target values when deriving category representations.
• labeluses · A label is a target value, category, or reference outcome associated with an example and used for learning, prediction, or evaluation.
• embeddingalternative to · An embedding is a numerical vector representation of an item that encodes features or relationships useful for computational tasks.
• high-cardinality categorical featureaddresses · A high-cardinality categorical feature is a categorical feature with many distinct values relative to its dataset or analysis context.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.