Data leakage

As explained in Graph Algorithms for Data Science

Data leakage is the incorporation of information into model features or training that would not be available when making the intended prediction.

Data leakage happens when training data includes information about the prediction outcome that will not be available when predictions are made. It can produce impressive training and evaluation scores while failing on genuinely new cases. In link prediction, leakage is especially easy to introduce because graph features are computed from relationships that may also define the labels the classifier must predict. section 10.1 section 10.2

Feature leakage is a direct form of data leakage: a feature contains the same or very similar information as the output variable. Treat it as a model evaluation shortcut rather than useful predictive signal. If a classifier can recover the answer from a feature that would not exist at prediction time, its measured accuracy does not represent its ability to predict future or missing links. section 10.1 section 10.2.2

Shortest-path leakage

Suppose network distance is used to predict whether two nodes have a relationship. If that distance is calculated from the same relationships used as positive training and test examples, directly linked pairs have distance 1. The model can then learn that one-hop pairs always have a link, yielding perfect-looking training and test accuracy while performing poorly on missing or future links. section 10.1 section 10.2.2

Figure 10.5 Building a link prediction model to recommend Twitch channels

The workflow separates relationships for feature generation, model training, and evaluation so graph-derived features cannot reveal the supervised link label. section 10.1

Prevent leakage in graph-based link prediction by separating the relationship data into distinct roles:

• Use one relationship set only to construct network features, such as distances or similarity measures. section 10.1 section 10.2
• Use a different relationship set as supervised samples for training the classification model. section 10.2
• Reserve another relationship set to evaluate the trained model. section 10.2
• Optionally add a validation set when performing hyperparameter optimization. section 10.2.2

This separation matters for many graph features, not only shortest-path distance. Features based on node closeness, similarity, or homophily-based node embeddings can leak label information when calculated from an improper split. Calculate features for both training and test samples strictly from the designated feature relationships; do not let either supervised set alter the graph used to derive those features. section 10.2 section 10.3

The split strategy depends on the prediction task. A random relationship split fits link completion when no time information is available. It randomly assigns relationships to feature, training, and evaluation roles while preserving the central rule: links used for feature calculation must differ from supervised links used to train and evaluate the classifier. section 10.2.2

Twitch shared-audience links

For a shared-audience network, a random query can mark a subset of Stream-to-Stream relationships as FEATURE_REL. Network features are then derived from those marked links, while separate relationship types hold positive and negative supervised samples. Keeping these roles apart supports recommendations for likely future audience overlap rather than merely rediscovering existing links. section 10.2.2 section 10.3

Leakage prevention also shapes feature choices. Manually defined link-prediction features can be evaluated first, while learned embeddings may require operational trade-offs. Transductive node embeddings used for link features can require retraining when a new node enters the graph; more complex inductive embeddings are a later option if needed. section 10.3

How Graph Algorithms for Data Science covers it

How deep it goes

Explains leakage through shortest-path examples and a three-way relationship split, without detailing classifier training internals.

Examples

Demonstrates leakage with one-hop network distance and a random split of Twitch shared-audience relationships.

What it assumes

Uses a Neo4j-loaded Twitch dataset and a Jupyter notebook for the random-split implementation.

What you can do after

Apply separate feature, training, and evaluation relationship sets to avoid leakage in graph-based link prediction.

How it connects

Connects data leakage with feature leakage, link prediction, network distance, node embeddings, and hyperparameter optimization.

liveBookmanning.com

Read Graph Algorithms for Data Science on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIInformation leakage, or data leakage, is the use or transfer of predictive information unavailable at prediction time, producing overly optimistic evaluation results.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningData leakage is the unintended use of information during model training or preprocessing that would not be available when making the intended predictions.
• 

  liveBookmanning.com
  Delivering Machine Learning Projects: From design to deploymentData leakage is the unintended use or transfer of information from evaluation data or future observations into model training or selection, causing misleadingly optimistic performance estimates.
• 

  liveBookmanning.com
  LLMs in ProductionData leakage is the inclusion of evaluation or otherwise unavailable information in a model's training data or training process.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA data-preparation failure in which label-related information incorrectly appears in features and inflates apparent model performance.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAn information leak is unintended transfer of information from evaluation data into a model or its development decisions.
• 

  liveBookmanning.com
  Machine Learning Platform EngineeringData leakage is the unintended inclusion or reuse of information in a dataset that compromises valid model development or evaluation.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionThe unintended transfer of information from evaluation data into model choices, causing biased estimates of generalization.

Compare how 9 books explain data leakage →

Related concepts

• dataset splitaddressed by · A dataset split is the division of a dataset into distinct subsets assigned different purposes, such as training, validation, or testing.
• defense in depthaddressed by · Defense in depth is a security strategy that combines multiple protective controls so that the failure or compromise of one does not eliminate all protection.

This explainer was generated by Manning's knowledge graph from the text of Graph Algorithms for Data Science. The section links point to the parts of the book it draws on.