liveBook · Manning

Data leakage

As explained in Build AI Drug Discovery Pipelines

Data leakage is the inadvertent use of information outside a training partition during model fitting, producing biased performance estimates.

Data leakage occurs when information from outside the training partition influences model fitting. It makes validation scores look better than the model’s reliability on unseen compounds. section 3.4.3

Common leakage patterns include:

  • Scaling features with a mean calculated from the complete dataset before separating training and test data lets test-set information influence training. section 3.4.3
  • Selecting features after exploratory analysis of the complete dataset can use test-data patterns and contaminate later evaluation. section 3.4.3
  • Using a biologically meaningful column such as pIC50 as a feature would undermine prediction of the related hERG-blocker class; pIC50 is reserved for understanding the dataset. section 3.2.1

Prevent leakage by placing preprocessing, molecular standardization, feature generation, and the estimator inside one scikit-learn pipeline. During cross-validation, the full sequence is refit independently within each fold, so transformations learn only from that fold’s training data. GridSearchCV or RandomizedSearchCV can then tune the complete workflow while preserving this separation. section 3.4.3

Leakage-safe ligand classification

A pipeline can convert SMILES to standardized molecules, generate fingerprints, and fit an SGDClassifier. GridSearchCV evaluates alternatives such as fingerprint radius, bit count, penalty, and alpha with five-fold cross-validation, producing a selected estimator and cross-validated F1 score.

Listing 3.14 Pipeline with GridSearchCV
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.pipeline import make_pipeline, Pipeline
from sklearn.preprocessing import PolynomialFeatures
 
cache_dir = "gs_linear_sgd"
pipe_linear_sgd = make_pipeline(
  SmilesToMols(),  # Transformer: Load & standardize SMILES
  FingerprintFeaturizer(),  # Transformer: Feature generation step
  SGDClassifier(),  # Estimator: linear model via SGD
  memory=cache_dir
)
 
param_grid_linear_sgd = [{
  'sgdclassifier__penalty': ["l2", "l1"],
  'sgdclassifier__alpha': [1e-3, 1e-2, 1e-1],
  'fingerprintfeaturizer__radius': [2, 4],
  'fingerprintfeaturizer__nBits': [1024, 2048],
}]
 
gs_linear_sgd = GridSearchCV(
  estimator=pipe_linear_sgd,
  param_grid=param_grid_linear_sgd,
  scoring="f1",
  cv=5,
  verbose=1,
)
 
gs_linear_sgd.fit(train_set.SMILES, train_set.Class)

Inspect the complete pipeline and parameter grid used to perform leakage-safe hyperparameter search. section 3.4.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines leakage, demonstrates scaling and feature-selection failures, and stops short of broader dataset-splitting strategies.
Examples
Uses feature scaling, exploratory feature selection, pIC50 exclusion, SMILES transformers, fingerprints, and GridSearchCV.
What it assumes
Assumes basic training, test partitions, features, labels, and cross-validation while defining leakage through molecular examples.
What you can do after
Apply via scikit-learn pipelines that refit preprocessing within each cross-validation fold.
How it connects
Connects data preprocessing, feature selection, cross-validation, hyperparameter tuning, pipelines, and test sets.

Also explained in

Compare how 12 books explain data leakage →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.