Model calibration
Model calibration is the assessment or adjustment of a model’s outputs, scores, or confidence so they correspond more reliably to observed outcomes or a reference standard.
Build AI Drug Discovery Pipelines
Calibration is the agreement between a model's predicted probabilities and the observed frequencies of outcomes among examples receiving those probabilities.
- How it is used
- Supports reliable probability thresholds for drug-discovery decisions about inhibition and toxicity.
- How deep it goes
- Explains reliability diagrams and error metrics, then demonstrates two post-training calibration methods with separate validation and test data.
- Examples
- Uses CYP3A4 inhibitor predictions and a toxicity-screening scenario to illustrate probability reliability.
- What you can do after
- Apply it via scikit-learn to fit sigmoid or isotonic calibrators and inspect test-set reliability diagrams.
Read how Build AI Drug Discovery Pipelines explains model calibration →
LLM Evaluation and Alignment, The Foundational Ideas
Calibration is the adjustment of a model's confidence or uncertainty expression so that it appropriately reflects the reliability of its outputs.
- How it is used
- Aligns confidence language, judge outputs, and detection decisions with evidence or human judgments.
- How deep it goes
- Provides calibration procedures for judge agreement, score compression, and detection thresholds, alongside a testable confidence-language rule.
- Examples
- Uses a financial research assistant’s confidence rules and application-specific hallucination detector thresholds.
- What you can do after
- Operate and configure judges and detectors using human-labeled calibration sets, agreement checks, score monitoring, and threshold selection.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains model calibration →
AI Model Evaluation
Calibration is the agreement between a model's predicted confidence and the observed frequency of its outcomes.
- How it is used
- Supports reliable LLM judging by anchoring scores to human standards and checking confidence before tiered routing.
- How deep it goes
- Provides gold-example, rerun, prompt-iteration, and confidence-stratification checks, while distinguishing calibration from full validation.
- Examples
- Uses a movie recommendation judge with human-scored examples and an offline comparison of first- and second-tier judges.
- What you can do after
- Evaluate and understand whether judge scores align with gold examples and whether confidence levels predict correctness.
Retrieval Augmented Generation, The Foundational Ideas
Calibration is the degree to which a model's confidence or categorical assessments agree with externally observed outcomes or labels.
- How it is used
- Measures whether confidence signals or retrieval judgments align with external labels, supporting monitoring and retrieval correction.
- How deep it goes
- Compares reflection tokens with external labels using classification and rank metrics, then diagnoses retrieval threshold and evaluator failures.
- Examples
- Uses Self-RAG support judgments checked against human labels and CRAG retrieval thresholds applied to stale policy passages.
- What you can do after
- Diagnose/debug with it: compare reflection tokens against external labels, or adjust retrieval thresholds and evaluator prompts.
Read how Retrieval Augmented Generation, The Foundational Ideas explains model calibration →
Timeless Algorithms, The Foundational Ideas
Probability calibration adjusts model scores so that predicted probabilities correspond to observed outcome frequencies.
- How it is used
- Uses probability reliability to inform thresholds, resource allocation, risk assessments, and operational actions.
- How deep it goes
- Defines calibration through predicted and observed frequencies but does not specify a correction procedure.
- Examples
- Contrasts 80% predictions followed by 80% correctness with the same predictions followed by only 60% correctness.
- What you can do after
- Evaluate whether predicted probabilities reliably correspond to observed outcome frequencies.
Read how Timeless Algorithms, The Foundational Ideas explains model calibration →
AI Agents in Action, Second Edition
Confidence calibration is the degree to which an agent’s expressed or inferred confidence corresponds to the correctness of its output.
- How it is used
- Evaluates whether an agent’s confidence signals track answer correctness during metacognitive monitoring and agent comparisons.
- How deep it goes
- Defines calibration, contrasts implicit signals with verbal claims, and outlines correlation and calibration-curve evaluation without detailing a calibration algorithm.
- Examples
- Uses ten varied-difficulty queries with confidence scores and ground-truth correctness, comparing correlation and confidence-bucket accuracy against a flat ReAct agent.
- What you can do after
- Evaluate an agent by recording confidence and correctness, computing their correlation, plotting a calibration curve, and comparing a ReAct baseline.
Read how AI Agents in Action, Second Edition explains model calibration →
Build an AI Agent (From Scratch)
Calibration is the agreement between a model's confidence or solvability judgments and the correctness of those judgments.
- How it is used
- Uses solvability judgments to indicate when an agent needs external tools rather than relying on uncertain model knowledge.
- How deep it goes
- Defines calibration through evaluation results and model behavior, without detailing a calibration procedure.
- Examples
- Compares four models’ accuracy with their solvability judgments across 20 problems.
- What you can do after
- Evaluate model self-assessment by comparing judged solvability with observed accuracy.
Read how Build an AI Agent (From Scratch) explains model calibration →
Also covered in
- Think Like a Software Engineering Manager
- Digital Twins in Action
- Vibe EngineeringIn early access (MEAP)
- Rearchitecting LLMsIn early access (MEAP)
- Designing AI AgentsIn early access (MEAP)
What to know first
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- precisionPrecision is the proportion of items predicted, selected, or retrieved as positive that are correct or relevant.
- recallRecall is the proportion of all relevant or actual positive items that a system successfully retrieves or correctly identifies.
- hallucinationA hallucination is model-generated content that is false, fabricated, or unsupported by relevant evidence but presented as plausible or factual.
Related concepts
- benchmarkmeasured by · A benchmark is a defined, repeatable task, workload, test, dataset, or reference standard used to measure or compare systems or methods under specified conditions.
- gold exampleuses · A gold example is an evaluation case whose expected judgment or score has been established by human experts or another trusted reference process.
- metacognitionpart of · Metacognition is the capacity to monitor, evaluate, or regulate one's own cognitive processes.
- validationcontrasts with · Validation is the process of checking data, behavior, or results against defined criteria to determine their correctness, acceptability, or suitability for a purpose.
- alignmentaddresses · Alignment is the state or process of bringing elements into correspondence with a shared reference, constraint, objective, or direction.
Questions about model calibration
What is model calibration?
Model calibration is the assessment or adjustment of a model’s outputs, scores, or confidence so they correspond more reliably to observed outcomes or a reference standard.
Which Manning books explain model calibration?
Build AI Drug Discovery Pipelines, LLM Evaluation and Alignment, The Foundational Ideas, AI Model Evaluation, Retrieval Augmented Generation, The Foundational Ideas, Timeless Algorithms, The Foundational Ideas, AI Agents in Action, Second Edition and Build an AI Agent (From Scratch).
What should I know before learning model calibration?
Manning books that teach model calibration usually introduce accuracy, precision, recall and hallucination first.
What is the difference between model calibration and validation?
Model calibration is the assessment or adjustment of a model’s outputs, scores, or confidence so they correspond more reliably to observed outcomes or a reference standard. Validation is the process of checking data, behavior, or results against defined criteria to determine their correctness, acceptability, or suitability for a purpose.
What is model calibration part of?
Model calibration is part of metacognition. Metacognition is the capacity to monitor, evaluate, or regulate one's own cognitive processes.
What is model calibration also called?
Model calibration is also called confidence calibration or probability calibration.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.






