Model calibration is the agreement between predicted confidence and observed outcomes. For an LLM judge, a confidence label is useful for routing decisions only if it predicts correctness: high-confidence verdicts should be correct more often than lower-confidence ones, and the stated confidence should reflect observed accuracy. Accuracy, consistency, and confidence calibration are distinct properties. section 10.5.1
Check the signal before scaling a tiered judging system. Run the first- and second-tier judges on the same offline set, then measure first-tier accuracy by its self-reported confidence level. If high-confidence verdicts are wrong nearly as often as medium-confidence verdicts, adjusting routing thresholds will not make the confidence label a useful signal. section 10.5.1
Calibration can also refer to anchoring an LLM judge’s scores with labeled examples. This few-shot calibration teaches the judge what scores such as 5 and 2 look like under a particular rubric. It can improve alignment with human standards, but it is not the same as measuring whether confidence predicts correctness. section 9.6.1 section 10.5.1
Before running an LLM judge at scale:
- Compare judgments with human-labeled gold examples, and use representative scored cases to anchor the rubric. section 9.10.3 section 9.6.1
- Repeat examples to spot variance, then revise the prompt and check consistency across random seeds or few-shot contexts. section 9.10.3
- Validate reliability separately against human ratings, gold labels, known examples, or canary sets. Anchoring scores alone is not full validation. section 9.11
- Log calibration or human-alignment results alongside the prompt and model versions, rubric, sampling strategy, and dataset snapshot so the run can be traced and reproduced. section 9.10.2
Calibrating a movie recommendation judge
A judge scores recommendations on criteria such as diversity and discovery. Give it a few recommendations pre-scored by humans or domain experts to illustrate the expected reasoning and score distribution. Then compare prompt versions using stability across reruns or correlation with human ratings; do not treat the resulting score as objective truth without validation. section 9.6.1 section 9.11






