liveBook · Manning

Miscalibration

As explained in Build AI Drug Discovery Pipelines

Miscalibration is the discrepancy between a model's predicted probabilities and the observed frequencies of the corresponding outcomes.

Miscalibration is a mismatch between predicted probabilities and observed outcome frequencies. For a well-calibrated inhibitor model, roughly 80% of molecules assigned probabilities near 0.8 should actually be inhibitors. The probability describes a group of similar predictions, not certainty about one molecule. section 5.1.3

A sigmoid function puts model outputs between 0 and 1, but does not guarantee that they match observed frequencies. In drug discovery, an inflated toxicity probability could lead researchers to discard a promising compound. Decision thresholds are more meaningful when the probabilities are calibrated. section 5.1.3

A practical calibration workflow:

  • Group predictions into probability bins and compare each bin’s predicted probabilities with its observed outcomes using a reliability diagram. section 5.1.3
  • Use root mean squared calibration error and Brier score alongside the diagram to assess predictions. section 5.5
  • Fit a post-training calibrator, such as Platt scaling or isotonic regression, on validation data; then inspect its predictions on a separate test set. section 5.1.3 section 5.5
Listing 5.2 Calibration with Platt Scaling & Isotonic Regression
from sklearn.calibration import CalibratedClassifierCV
from sklearn.frozen import FrozenEstimator
from utils import reliability_diagram
 
lr_ps_cal = CalibratedClassifierCV(
    FrozenEstimator(lr_model), method='sigmoid'
)
lr_ps_cal.fit(val_fingerprints, val_df.Inhibitor)
test_proba_ps = lr_ps_cal.predict_proba(test_fingerprints)[:, 1]
ax, ax1 = reliability_diagram(test_proba_ps, test_df.Inhibitor, nbins=10)
 
lr_ir_cal = CalibratedClassifierCV(
    FrozenEstimator(lr_model), method='isotonic'
)
lr_ir_cal.fit(val_fingerprints, val_df.Inhibitor)
test_proba_ir = lr_ir_cal.predict_proba(test_fingerprints)[:, 1]
ax, ax1 = reliability_diagram(test_proba_ir, test_df.Inhibitor, nbins=10)

This listing calibrates a fitted logistic regression model with sigmoid and isotonic methods, then plots test-set reliability diagrams. section 5.1.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains probability interpretation, reliability diagrams and diagnostic metrics, then demonstrates post-training calibration on validation and test data.
Examples
Uses cytochrome P450 inhibition predictions and a toxicity-screening scenario to illustrate calibration and its practical stakes.
What you can do after
Diagnose miscalibration with reliability diagrams and apply post-training calibration using scikit-learn.
How it connects
Connects miscalibration to logistic regression, reliability diagrams, decision thresholds, Platt scaling and isotonic regression.

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.