liveBook · Manning

Model calibration

As explained in Build AI Drug Discovery Pipelines

Calibration is the agreement between a model's predicted probabilities and the observed frequencies of outcomes among examples receiving those probabilities.

Model calibration describes whether predicted probabilities agree with observed outcome frequencies. For a well-calibrated inhibitor model, roughly 80% of molecules assigned a probability near 0.8 should actually be inhibitors. The probability describes a group of similar predictions, not a guarantee about one molecule. section 5.1.3

A sigmoid function puts a model’s output between 0 and 1, but that alone does not make the result a reliable probability. Miscalibration can make decision thresholds misleading: an inflated toxicity estimate, for example, could cause a promising compound to be rejected. section 5.1.3

Use a reliability diagram to assess calibration:

  • Group predictions into probability bins, such as 0–10%, 10–20%, and so on. section 5.1.3
  • Compare the predictions in each bin with the observed frequency of the outcome. Large gaps indicate miscalibration. section 5.1.3
  • Summarize discrepancies with a calibration error metric, or use the Brier score to measure squared differences between individual probabilities and outcomes. section 5.1.3
Figure 5.2 Reliability diagram with error bars for logistic regressor trained on CYP 3A4 data set. Probability estimates, RMSCE, and BS derived from predictions on validation set.

This reliability diagram shows validation-set probability estimates for a logistic regressor predicting CYP3A4 inhibition, with error bars and summary scores. section 5.1.3

Calibration can also be applied after training. Platt scaling and isotonic regression fit a mapping using labeled validation predictions. Evaluate the resulting probabilities on a separate test set rather than the data used to fit the mapping. section 5.1.3

Listing 5.2 Calibration with Platt Scaling & Isotonic Regression
from sklearn.calibration import CalibratedClassifierCV
from sklearn.frozen import FrozenEstimator
from utils import reliability_diagram
 
lr_ps_cal = CalibratedClassifierCV(
    FrozenEstimator(lr_model), method='sigmoid'
)
lr_ps_cal.fit(val_fingerprints, val_df.Inhibitor)
test_proba_ps = lr_ps_cal.predict_proba(test_fingerprints)[:, 1]
ax, ax1 = reliability_diagram(test_proba_ps, test_df.Inhibitor, nbins=10)
 
lr_ir_cal = CalibratedClassifierCV(
    FrozenEstimator(lr_model), method='isotonic'
)
lr_ir_cal.fit(val_fingerprints, val_df.Inhibitor)
test_proba_ir = lr_ir_cal.predict_proba(test_fingerprints)[:, 1]
ax, ax1 = reliability_diagram(test_proba_ir, test_df.Inhibitor, nbins=10)

This Python listing freezes a trained logistic regressor, fits sigmoid and isotonic calibrators on validation data, and plots their test-set reliability diagrams. section 5.1.3

Figure 5.3 Reliability diagram for the logistic regressor after undergoing calibration via platt scaling or isotonic regression.

Compare the reliability diagrams after Platt scaling and isotonic regression. section 5.1.3

In multiclass or multilabel prediction, different classes or labels can have different calibration and optimal thresholds. These differences can affect the ranking of labels, especially with independently trained models, more complex models, or greater data imbalance. section 5.4.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains reliability diagrams and error metrics, then demonstrates two post-training calibration methods with separate validation and test data.
Examples
Uses CYP3A4 inhibitor predictions and a toxicity-screening scenario to illustrate probability reliability.
What you can do after
Apply it via scikit-learn to fit sigmoid or isotonic calibrators and inspect test-set reliability diagrams.
How it connects
Connects calibration to classification thresholds, logistic regression, label ranking, and class imbalance.

Also explained in

Compare how 7 books explain model calibration →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.