Miscalibration is a mismatch between predicted probabilities and observed outcome frequencies. For a well-calibrated inhibitor model, roughly 80% of molecules assigned probabilities near 0.8 should actually be inhibitors. The probability describes a group of similar predictions, not certainty about one molecule. section 5.1.3
A sigmoid function puts model outputs between 0 and 1, but does not guarantee that they match observed frequencies. In drug discovery, an inflated toxicity probability could lead researchers to discard a promising compound. Decision thresholds are more meaningful when the probabilities are calibrated. section 5.1.3
A practical calibration workflow:
- Group predictions into probability bins and compare each bin’s predicted probabilities with its observed outcomes using a reliability diagram. section 5.1.3
- Use root mean squared calibration error and Brier score alongside the diagram to assess predictions. section 5.5
- Fit a post-training calibrator, such as Platt scaling or isotonic regression, on validation data; then inspect its predictions on a separate test set. section 5.1.3 section 5.5
Listing 5.2 Calibration with Platt Scaling & Isotonic Regression
from sklearn.calibration import CalibratedClassifierCV
from sklearn.frozen import FrozenEstimator
from utils import reliability_diagram
lr_ps_cal = CalibratedClassifierCV(
FrozenEstimator(lr_model), method='sigmoid'
)
lr_ps_cal.fit(val_fingerprints, val_df.Inhibitor)
test_proba_ps = lr_ps_cal.predict_proba(test_fingerprints)[:, 1]
ax, ax1 = reliability_diagram(test_proba_ps, test_df.Inhibitor, nbins=10)
lr_ir_cal = CalibratedClassifierCV(
FrozenEstimator(lr_model), method='isotonic'
)
lr_ir_cal.fit(val_fingerprints, val_df.Inhibitor)
test_proba_ir = lr_ir_cal.predict_proba(test_fingerprints)[:, 1]
ax, ax1 = reliability_diagram(test_proba_ir, test_df.Inhibitor, nbins=10)This listing calibrates a fitted logistic regression model with sigmoid and isotonic methods, then plots test-set reliability diagrams. section 5.1.3
