Root mean squared calibration error (RMSCE) measures how closely predicted probabilities match observed outcome frequencies across probability bins. In cytochrome P450 inhibition classification, it helps assess whether predicted risks are trustworthy for decisions involving drug-drug interactions. A value of 0 represents perfectly scaled probabilities; lower values are better. section 5.1.3 section 5.5
Interpret RMSCE alongside reliability diagnostics rather than treating it as a standalone model score:
- Compare each bin's predicted probability with its empirical outcome frequency; points above the diagonal indicate underconfidence, while points below it indicate overconfidence. section 5.1.3
- Use RMSCE to summarize alignment across bins when comparing many models, but remember that the binning strategy and data concentration can change the metric. section 5.1.3
- Pair RMSCE with Brier score when binning sensitivity matters; Brier score compares predictions with observed binary outcomes at the instance level. section 5.1.3
Checking a logistic regressor
A logistic regressor trained on CYP 3A4 data is evaluated with a validation-set reliability diagram, RMSCE, and Brier score. Its points remain near the diagonal across most probability bins, with modest overconfidence and underconfidence, indicating decent overall calibration. section 5.1.3
Use the equation to calculate the root mean squared difference between each bin's predicted probability and observed frequency, then summarize calibration error across the bins. section 5.1.3
