Data leakage

As explained in Ensemble Methods for Machine Learning

Data leakage is the unintended use of information during model training or preprocessing that would not be available when making the intended predictions.

Data leakage occurs when information that should be unavailable at prediction time influences training, preprocessing, validation, or testing. It makes evaluation look stronger than likely deployment performance. A hold-out test set is meant to simulate unseen data, so it must remain separate from every modeling stage until final evaluation. section 8.1.3

Two forms matter when preprocessing categorical features. Feature leakage lets feature information cross from training into the test set. Target leakage lets labels cross that boundary. Target encoding is especially risky because it replaces a category with a statistic, such as the mean target value for rows in that category. If labels are used carelessly to produce encodings, the model can indirectly receive information it should not have. section 8.1.3

Figure 8.3 Target leakage from the training to test set illustrated. All the targets (labels) in the training set are greedily used to create an encoding for red, which is used to encode this category in both the training and test sets, leading to target leakage.

This illustration shows target leakage: labels from the training partition determine the encoding for red, and that encoding is then used across both partitions. section 8.1.3

Guard against leakage by preserving the intended train/test boundary throughout preprocessing and evaluation:

• Split features and labels into training and test data before fitting transformations. section 8.1.4
• Fit a preprocessing transformer on training features, then transform test features with that fitted transformer rather than fitting it again. section 8.1.4
• Use pipelines to combine preprocessing and modeling consistently, including imputation, scaling, and categorical encoding. section 8.3.2
• Keep the final test set for final performance evaluation while using cross-validation and parameter search during model selection. section 8.3.2

Greedy target encoding uses all available training labels to compute category statistics. Although it can yield a compact representation, it can leak target information and can also create distribution shift between encoded training and test data. Hold-out encoding avoids both problems by reserving a third partition exclusively for target-statistic encoding. Leave-one-out encoding avoids reserving that extra subset by excluding the individual training example being encoded; test data is encoded with greedy target encoding. section 8.1.3

Figure 8.5 Hold-out encoding partitions the available data into three sets: training and test, as usual, and a third hold-out test set to be used exclusively for encoding with target statistics. This avoids both target leakage and distribution shift.

Hold-out encoding adds a dedicated encoding partition, separating the labels used for target statistics from ordinary training and test roles. section 8.1.3

Categorical credit-approval pipeline

For binary credit approval, continuous columns can be standardized while categorical columns are encoded through separate pipelines joined by a ColumnTransformer. The transformer is fit on training data and applied to test data. One-hot encoding expanded the representation to 38 features and produced 89.855% test accuracy in the shown run; greedy target encoding used 14 features and produced 91.304%, while leave-one-out target encoding used 14 features and produced 90.580%. These results compare encoders, but the pipeline boundary is what keeps preprocessing aligned with evaluation. section 8.1.4

How Ensemble Methods for Machine Learning covers it

How deep it goes

Defines feature and target leakage, then contrasts greedy, hold-out, and leave-one-out target encoding procedures.

Examples

Uses color-category target statistics and Australian Credit Approval encoding pipelines with one-hot, greedy target, and leave-one-out encoders.

What it assumes

Assumes familiarity with train-test splits, categorical features, labels, and basic model evaluation.

What you can do after

Apply it via scikit-learn-compatible preprocessing pipelines that fit transformations on training data and evaluate final models on held-out data.

How it connects

Connects target encoding with distribution shift, hold-out test sets, preprocessing pipelines, cross-validation, and model evaluation.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIInformation leakage, or data leakage, is the use or transfer of predictive information unavailable at prediction time, producing overly optimistic evaluation results.
• 

  liveBookmanning.com
  Graph Algorithms for Data ScienceData leakage is the incorporation of information into model features or training that would not be available when making the intended prediction.
• 

  liveBookmanning.com
  Delivering Machine Learning Projects: From design to deploymentData leakage is the unintended use or transfer of information from evaluation data or future observations into model training or selection, causing misleadingly optimistic performance estimates.
• 

  liveBookmanning.com
  LLMs in ProductionData leakage is the inclusion of evaluation or otherwise unavailable information in a model's training data or training process.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA data-preparation failure in which label-related information incorrectly appears in features and inflates apparent model performance.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAn information leak is unintended transfer of information from evaluation data into a model or its development decisions.
• 

  liveBookmanning.com
  Machine Learning Platform EngineeringData leakage is the unintended inclusion or reuse of information in a dataset that compromises valid model development or evaluation.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionThe unintended transfer of information from evaluation data into model choices, causing biased estimates of generalization.

Compare how 9 books explain data leakage →

Related concepts

• dataset splitaddressed by · A dataset split is the division of a dataset into distinct subsets assigned different purposes, such as training, validation, or testing.
• defense in depthaddressed by · Defense in depth is a security strategy that combines multiple protective controls so that the failure or compromise of one does not eliminate all protection.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.