liveBook · Manning

Data leakage

As explained in Inside Deep Learning: Math, Algorithms, Models

A data-preparation failure in which label-related information incorrectly appears in features and inflates apparent model performance.

Data leakage occurs when label-related information enters the input features through an unrealistic artifact, often during data preparation. It inflates apparent performance and can taint every model built from the data. section 12.2

Investigate suspiciously strong performance by:

  • Questioning whether an innocuous feature, such as length, correlates unexpectedly with the goal. section 12.2
  • Thinking like a cheater and searching for shortcuts that reveal the label. section 12.2
  • Labeling data by hand and comparing human shortcuts with the model’s apparent biases. section 12.2
  • Fixing the leakage before continuing model development. section 12.2

Malware detection

A classifier reached near-perfect accuracy because benign Windows data commonly contained “Copyright Microsoft Corporation,” while malware data did not. The string exposed the benign label instead of representing a naturally valid predictive signal. section 12.2

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes
Defines label leakage, identifies preparation bugs and label inclusion, and illustrates one concrete case without cataloging every possible mechanism.
Examples
Uses unexpected feature correlations, manual labeling, and a Microsoft copyright string in malware detection.
What you can do after
Diagnose suspicious shortcuts and repair information leakage before building additional models.
How it connects
Connects labels, input features, data preparation, model performance, and model bias.

Also explained in

Compare how 12 books explain data leakage →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.