Model calibration
Model calibration is the assessment or adjustment of a model’s outputs, scores, or confidence so they correspond more reliably to observed outcomes or a reference standard.
AI Agents in Action, Second Edition
Confidence calibration is the degree to which an agent’s expressed or inferred confidence corresponds to the correctness of its output.
- How it is used
- Evaluates whether an agent’s confidence signals track answer correctness during metacognitive monitoring and agent comparisons.
- How deep it goes
- Defines calibration, contrasts implicit signals with verbal claims, and outlines correlation and calibration-curve evaluation without detailing a calibration algorithm.
- Examples
- Uses ten varied-difficulty queries with confidence scores and ground-truth correctness, comparing correlation and confidence-bucket accuracy against a flat ReAct agent.
- What you can do after
- Evaluate an agent by recording confidence and correctness, computing their correlation, plotting a calibration curve, and comparing a ReAct baseline.
Read how AI Agents in Action, Second Edition explains model calibration →
Build an AI Agent (From Scratch)
Calibration is the agreement between a model's confidence or solvability judgments and the correctness of those judgments.
- How it is used
- Uses solvability judgments to indicate when an agent needs external tools rather than relying on uncertain model knowledge.
- How deep it goes
- Defines calibration through evaluation results and model behavior, without detailing a calibration procedure.
- Examples
- Compares four models’ accuracy with their solvability judgments across 20 problems.
- What you can do after
- Evaluate model self-assessment by comparing judged solvability with observed accuracy.
Read how Build an AI Agent (From Scratch) explains model calibration →
Related concepts
- metacognitionpart of · Metacognition is the capacity to monitor, evaluate, or regulate one's own cognitive processes.
- validationcontrasts with · The process of checking data, behavior, systems, models, or results against defined criteria to assess their correctness, suitability, or acceptability.
- benchmarkmeasured by · A benchmark is a defined task, dataset, workload, procedure, or reference measure used to assess or compare performance or capability under specified criteria or conditions.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.