liveBook · Manning

Model calibration

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Calibration is the adjustment of a model's confidence or uncertainty expression so that it appropriately reflects the reliability of its outputs.

A model can sound certain even when its evidence is weak. Calibration aligns its expressions of confidence with the reliability of its outputs. It can also mean checking that an AI judge’s scores or preferences correspond to human judgments, or choosing a detection threshold that suits the cost of errors. section 8.5.1 section 4.4.4 section 5.4.3

For a financial research assistant, confidence can be specified as a testable behavior: hedge claims based on retrieved context, use confident language only for direct quotations, and avoid forward-looking statements without an explicit basis in the context. Fine-tuning on verified data is another approach to improving faithfulness and uncertainty expression, but detection remains necessary because mitigation does not eliminate hallucinations. section 8.5.1 section 5.4.5

Calibrating an AI judge requires a human reference and a clear decision about what its output will be used for:

  • Choose the scale. Binary pass/fail decisions are robust to ambiguous score boundaries but cannot distinguish quality within either category. Anchored rating scales make score meanings clearer; unanchored numeric scales can compress scores around the middle. section 4.4.2
  • Check preferences against people. Sample 500–1,000 prompts from the production distribution, generate two responses per prompt, and have two or three human labelers rank each pair. Compare AI–human agreement with human–human agreement using the same judge prompt and configuration intended for deployment. section 8.3.2
  • Investigate a substantial agreement gap. Vague principles or an insufficiently capable judge can cause the labeler to solve a different problem from the one intended. Reasoning or few-shot examples in the judge prompt can improve calibration. section 8.3.2
  • Monitor score distributions for drift or compression. For scalar ratings, map raw scores to human ratings from a 50–100 response calibration set. If necessary, use batch percentile ranks, which lose cross-batch comparability, or switch to a human-validated pass/fail threshold. section 4.4.4

Calibrating a hallucination detector

Annotate 100–200 responses for hallucinations, run the detector, and plot precision against recall at different thresholds. A medical application may favor recall and send more false alarms to human reviewers; an internal productivity tool may favor precision to avoid disrupting work. Recalibrate monthly or after a significant system change. section 5.4.3

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Provides calibration procedures for judge agreement, score compression, and detection thresholds, alongside a testable confidence-language rule.
Examples
Uses a financial research assistant’s confidence rules and application-specific hallucination detector thresholds.
What you can do after
Operate and configure judges and detectors using human-labeled calibration sets, agreement checks, score monitoring, and threshold selection.
How it connects
Connects calibration to hallucination detection, AI judges, precision and recall, and constitutional AI.

Also explained in

Compare how 7 books explain model calibration →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.