A reliability diagram, or reliability plot, compares predicted probabilities with observed outcome frequencies across probability bins. It assesses whether confidence estimates are calibrated: predictions near the diagonal align with real-world event rates, while large deviations indicate unreliable confidence. section 5.1.3
Construct a diagram by following this workflow:
- Divide predictions into probability bins, such as 0–10%, 10–20%, through 90–100%. section 5.1.3
- Plot each bin’s predicted probability on the x-axis and its observed outcome frequency on the y-axis. section 5.1.3
- Compare the points with the diagonal ideal line: a 20% prediction should correspond to approximately 20% positive outcomes. section 5.1.3
Listing 5.2 Calibration with Platt Scaling & Isotonic Regression
from sklearn.calibration import CalibratedClassifierCV
from sklearn.frozen import FrozenEstimator
from utils import reliability_diagram
lr_ps_cal = CalibratedClassifierCV(
FrozenEstimator(lr_model), method='sigmoid'
)
lr_ps_cal.fit(val_fingerprints, val_df.Inhibitor)
test_proba_ps = lr_ps_cal.predict_proba(test_fingerprints)[:, 1]
ax, ax1 = reliability_diagram(test_proba_ps, test_df.Inhibitor, nbins=10)
lr_ir_cal = CalibratedClassifierCV(
FrozenEstimator(lr_model), method='isotonic'
)
lr_ir_cal.fit(val_fingerprints, val_df.Inhibitor)
test_proba_ir = lr_ir_cal.predict_proba(test_fingerprints)[:, 1]
ax, ax1 = reliability_diagram(test_proba_ir, test_df.Inhibitor, nbins=10)A logistic regressor can be calibrated with Platt scaling or isotonic regression, then evaluated by generating test-set probabilities and plotting ten bins. section 5.1.3
CYP 3A4 inhibition
A reliability diagram for a logistic regressor trained on CYP 3A4 data uses validation-set probability estimates and includes error bars. Calibration quality can be summarized with RMSCE and Brier score; calibrated probabilities support thresholds for advancing, retesting, or eliminating compounds. section 5.1.3
