Target leaking

As explained in Ensemble Methods for Machine Learning

Target leakage is the inadvertent use of information derived from a prediction target in model inputs or training computations in a way that would not be available at prediction time.

Target leakage occurs when target or label information enters features, preprocessing, or training computations in a way unavailable when making predictions. It can make evaluation look better while weakening estimates of generalization. section 8.1.3

Greedy target encoding

Suppose the category red is replaced by the mean target for all red examples in the training data. Reusing that target-derived value to encode both training and test rows lets training labels influence test representations. This is target leakage and can also create prediction shift between the two sets. A hold-out encoding set separates target-statistic computation from ordinary training and testing; leave-one-out encoding instead excludes only the row currently being encoded, saving data but not completely removing prediction shift. section 8.1.3

Figure 8.3 Target leakage from the training to test set illustrated. All the targets (labels) in the training set are greedily used to create an encoding for red, which is used to encode this category in both the training and test sets, leading to target leakage.

Shows how training labels used to encode a category are carried into the test representation. section 8.1.3

Prevent or reduce leakage by applying the following controls:

• Keep the test set completely outside preprocessing, training, and validation; fit preprocessing on training data and only transform the test data afterward. section 8.1.3 section 8.1.4
• Use a hold-out encoding set when target statistics require strong separation, or use leave-one-out encoding when preserving training data matters more. section 8.1.3
• Use ordered target statistics when applying CatBoost: randomly permute training examples and compute each row’s statistics only from preceding rows. section 8.2.1
• Remember that boosting can create prediction shift through reused data even without categorical features; ordered boosting limits gradient information to preceding examples as well. section 8.2.1

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains leakage through greedy, hold-out, and leave-one-out target encoding, then connects it to ordered CatBoost methods.

Examples

Uses red-category target means, train-test partitions, hold-out encoding, leave-one-out encoding, and CatBoost permutations.

What it assumes

Assumes familiarity with training and test sets, preprocessing, categorical features, targets, and model evaluation.

What you can do after

Diagnose leakage by auditing test isolation and target-derived transformations, then apply hold-out, leave-one-out, or ordered encoding.

How it connects

Connects target leakage to data leakage, target encoding, prediction shift, preprocessing, CatBoost, and ordered boosting.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• mean target encodingproduced by · Mean target encoding is a categorical encoding technique that replaces each category with the mean of its associated target values.
• ordered boostingaddressed by · Ordered boosting is a boosting approach that uses ordered data processing to reduce prediction-shift bias during training.
• inputuses · An input is a value, information, or signal supplied to a system, computation, or interface for processing or to influence its behavior.
• hold-out target encodingaddressed by · Hold-out target encoding computes category encodings from a separate subset of data excluded from the examples being encoded.
• datasetuses · A dataset is an organized collection of related data items, records, or observations treated as a unit for storage, processing, analysis, or computation.
• training datasetaddressed by · A collection of examples or observations supplied to a learning procedure to fit or adapt a model or learn its patterns or parameters.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.