liveBook · Manning

Reliability diagram

As explained in Build AI Drug Discovery Pipelines

A reliability diagram, or reliability plot, compares predicted probabilities with observed outcome frequencies across probability bins to assess calibration.

A reliability diagram, or reliability plot, compares predicted probabilities with observed outcome frequencies across probability bins. It assesses whether confidence estimates are calibrated: predictions near the diagonal align with real-world event rates, while large deviations indicate unreliable confidence. section 5.1.3

Construct a diagram by following this workflow:

  • Divide predictions into probability bins, such as 0–10%, 10–20%, through 90–100%. section 5.1.3
  • Plot each bin’s predicted probability on the x-axis and its observed outcome frequency on the y-axis. section 5.1.3
  • Compare the points with the diagonal ideal line: a 20% prediction should correspond to approximately 20% positive outcomes. section 5.1.3
Listing 5.2 Calibration with Platt Scaling & Isotonic Regression
from sklearn.calibration import CalibratedClassifierCV
from sklearn.frozen import FrozenEstimator
from utils import reliability_diagram
 
lr_ps_cal = CalibratedClassifierCV(
    FrozenEstimator(lr_model), method='sigmoid'
)
lr_ps_cal.fit(val_fingerprints, val_df.Inhibitor)
test_proba_ps = lr_ps_cal.predict_proba(test_fingerprints)[:, 1]
ax, ax1 = reliability_diagram(test_proba_ps, test_df.Inhibitor, nbins=10)
 
lr_ir_cal = CalibratedClassifierCV(
    FrozenEstimator(lr_model), method='isotonic'
)
lr_ir_cal.fit(val_fingerprints, val_df.Inhibitor)
test_proba_ir = lr_ir_cal.predict_proba(test_fingerprints)[:, 1]
ax, ax1 = reliability_diagram(test_proba_ir, test_df.Inhibitor, nbins=10)

A logistic regressor can be calibrated with Platt scaling or isotonic regression, then evaluated by generating test-set probabilities and plotting ten bins. section 5.1.3

CYP 3A4 inhibition

A reliability diagram for a logistic regressor trained on CYP 3A4 data uses validation-set probability estimates and includes error bars. Calibration quality can be summarized with RMSCE and Brier score; calibrated probabilities support thresholds for advancing, retesting, or eliminating compounds. section 5.1.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines binning and the ideal diagonal, then connects diagnosis to RMSCE, ECE, MCE, Brier score, and post-training calibration.
Examples
Uses ten uniform inhibition-probability bins and CYP 3A4 logistic-regression predictions, including calibrated plots with error bars.
What it assumes
Assumes binary classification probabilities and observed labels, while defining binning, calibration, and confidence interpretation from basic examples.
What you can do after
Diagnose miscalibration, compare confidence with observed outcomes, and evaluate post-training calibration before applying decision thresholds.
How it connects
Connects probability calibration with logistic regression, decision thresholds, PR and ROC curves, Platt scaling, isotonic regression, and Brier score.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.