Confidence is a model-produced signal of how certain a prediction or evaluation appears. In judge-based systems, use it to decide when a cheap first-pass verdict needs stronger review, not as proof that the verdict is correct. section 10.5.2
Treat confidence as useful routing evidence, while checking its limits:
- Escalate low-confidence or borderline results, such as a score of 3 on a five-point scale when 3 separates pass from fail. section 10.5.2
- Escalate unstable results that change across repeated samples or under swapped-order testing. section 10.5.2
- Do not assume high confidence means correctness: a judge can confidently accept fluent prose containing a fabricated figure. section 10.5.2
A practical escalation rule
Route an evaluation to a stronger judge or human reviewer when the Tier 1 judge reports low confidence, produces a borderline score, varies across reruns, or reacts differently when answer order is swapped. section 10.5.2















