liveBook · Manning

Model calibration

Also called confidence calibration, probability calibration

Model calibration is the assessment or adjustment of a model’s outputs, scores, or confidence so they correspond more reliably to observed outcomes or a reference standard.

How 7 Manning books explain it

Build AI Drug Discovery Pipelines

Calibration is the agreement between a model's predicted probabilities and the observed frequencies of outcomes among examples receiving those probabilities.

How it is used
Supports reliable probability thresholds for drug-discovery decisions about inhibition and toxicity.
How deep it goes
Explains reliability diagrams and error metrics, then demonstrates two post-training calibration methods with separate validation and test data.
Examples
Uses CYP3A4 inhibitor predictions and a toxicity-screening scenario to illustrate probability reliability.
What you can do after
Apply it via scikit-learn to fit sigmoid or isotonic calibrators and inspect test-set reliability diagrams.

Read how Build AI Drug Discovery Pipelines explains model calibration →

LLM Evaluation and Alignment, The Foundational Ideas

Calibration is the adjustment of a model's confidence or uncertainty expression so that it appropriately reflects the reliability of its outputs.

How it is used
Aligns confidence language, judge outputs, and detection decisions with evidence or human judgments.
How deep it goes
Provides calibration procedures for judge agreement, score compression, and detection thresholds, alongside a testable confidence-language rule.
Examples
Uses a financial research assistant’s confidence rules and application-specific hallucination detector thresholds.
What you can do after
Operate and configure judges and detectors using human-labeled calibration sets, agreement checks, score monitoring, and threshold selection.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains model calibration →

AI Model Evaluation

Calibration is the agreement between a model's predicted confidence and the observed frequency of its outcomes.

How it is used
Supports reliable LLM judging by anchoring scores to human standards and checking confidence before tiered routing.
How deep it goes
Provides gold-example, rerun, prompt-iteration, and confidence-stratification checks, while distinguishing calibration from full validation.
Examples
Uses a movie recommendation judge with human-scored examples and an offline comparison of first- and second-tier judges.
What you can do after
Evaluate and understand whether judge scores align with gold examples and whether confidence levels predict correctness.

Read how AI Model Evaluation explains model calibration →

Retrieval Augmented Generation, The Foundational Ideas

Calibration is the degree to which a model's confidence or categorical assessments agree with externally observed outcomes or labels.

How it is used
Measures whether confidence signals or retrieval judgments align with external labels, supporting monitoring and retrieval correction.
How deep it goes
Compares reflection tokens with external labels using classification and rank metrics, then diagnoses retrieval threshold and evaluator failures.
Examples
Uses Self-RAG support judgments checked against human labels and CRAG retrieval thresholds applied to stale policy passages.
What you can do after
Diagnose/debug with it: compare reflection tokens against external labels, or adjust retrieval thresholds and evaluator prompts.

Read how Retrieval Augmented Generation, The Foundational Ideas explains model calibration →

Timeless Algorithms, The Foundational Ideas

Probability calibration adjusts model scores so that predicted probabilities correspond to observed outcome frequencies.

How it is used
Uses probability reliability to inform thresholds, resource allocation, risk assessments, and operational actions.
How deep it goes
Defines calibration through predicted and observed frequencies but does not specify a correction procedure.
Examples
Contrasts 80% predictions followed by 80% correctness with the same predictions followed by only 60% correctness.
What you can do after
Evaluate whether predicted probabilities reliably correspond to observed outcome frequencies.

Read how Timeless Algorithms, The Foundational Ideas explains model calibration →

AI Agents in Action, Second Edition

Confidence calibration is the degree to which an agent’s expressed or inferred confidence corresponds to the correctness of its output.

How it is used
Evaluates whether an agent’s confidence signals track answer correctness during metacognitive monitoring and agent comparisons.
How deep it goes
Defines calibration, contrasts implicit signals with verbal claims, and outlines correlation and calibration-curve evaluation without detailing a calibration algorithm.
Examples
Uses ten varied-difficulty queries with confidence scores and ground-truth correctness, comparing correlation and confidence-bucket accuracy against a flat ReAct agent.
What you can do after
Evaluate an agent by recording confidence and correctness, computing their correlation, plotting a calibration curve, and comparing a ReAct baseline.

Read how AI Agents in Action, Second Edition explains model calibration →

Build an AI Agent (From Scratch)

Calibration is the agreement between a model's confidence or solvability judgments and the correctness of those judgments.

How it is used
Uses solvability judgments to indicate when an agent needs external tools rather than relying on uncertain model knowledge.
How deep it goes
Defines calibration through evaluation results and model behavior, without detailing a calibration procedure.
Examples
Compares four models’ accuracy with their solvability judgments across 20 problems.
What you can do after
Evaluate model self-assessment by comparing judged solvability with observed accuracy.

Read how Build an AI Agent (From Scratch) explains model calibration →

Also covered in

What to know first

Related concepts

Questions about model calibration

What is model calibration?

Model calibration is the assessment or adjustment of a model’s outputs, scores, or confidence so they correspond more reliably to observed outcomes or a reference standard.

Which Manning books explain model calibration?

Build AI Drug Discovery Pipelines, LLM Evaluation and Alignment, The Foundational Ideas, AI Model Evaluation, Retrieval Augmented Generation, The Foundational Ideas, Timeless Algorithms, The Foundational Ideas, AI Agents in Action, Second Edition and Build an AI Agent (From Scratch).

What should I know before learning model calibration?

Manning books that teach model calibration usually introduce accuracy, precision, recall and hallucination first.

What is the difference between model calibration and validation?

Model calibration is the assessment or adjustment of a model’s outputs, scores, or confidence so they correspond more reliably to observed outcomes or a reference standard. Validation is the process of checking data, behavior, or results against defined criteria to determine their correctness, acceptability, or suitability for a purpose.

What is model calibration part of?

Model calibration is part of metacognition. Metacognition is the capacity to monitor, evaluate, or regulate one's own cognitive processes.

What is model calibration also called?

Model calibration is also called confidence calibration or probability calibration.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.