liveBook · Manning

Model calibration

As explained in Timeless Algorithms, The Foundational Ideas

Probability calibration adjusts model scores so that predicted probabilities correspond to observed outcome frequencies.

Calibration measures whether predicted probabilities match observed outcome frequencies. If a classifier assigns many cases a probability near 0.80, it is well calibrated when about 80% of those cases are correct: P(correct | predicted probability ≈ 0.80) ≈ 0.80. section 11.6.4

A probability is not a guarantee about an individual prediction. Calibration matters when probabilities guide thresholds, resource allocation, risk assessments, or operational actions. Similar classification accuracy does not ensure similar probability reliability; standard neural-network probabilities also do not automatically capture uncertainty across plausible parameter values. section 11.6.4

Checking an 0.80 prediction

If a model repeatedly assigns cases probabilities near 0.80 but is correct for only 60% of them, its probabilities are poorly calibrated. section 11.6.4

An SVM produces a decision score rather than a probability unless an additional calibration step is applied. section 8.8.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines calibration through predicted and observed frequencies but does not specify a correction procedure.
Examples
Contrasts 80% predictions followed by 80% correctness with the same predictions followed by only 60% correctness.
What you can do after
Evaluate whether predicted probabilities reliably correspond to observed outcome frequencies.
How it connects
Links probability calibration to classifier confidence, accuracy, and decision scores.

Also explained in

Compare how 7 books explain model calibration →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.