liveBook · Manning

Confidence

As explained in AI Model Evaluation

A numerical value produced by a model that represents its estimated certainty in an output or prediction.

Confidence is a model-produced signal of how certain a prediction or evaluation appears. In judge-based systems, use it to decide when a cheap first-pass verdict needs stronger review, not as proof that the verdict is correct. section 10.5.2

Treat confidence as useful routing evidence, while checking its limits:

  • Escalate low-confidence or borderline results, such as a score of 3 on a five-point scale when 3 separates pass from fail. section 10.5.2
  • Escalate unstable results that change across repeated samples or under swapped-order testing. section 10.5.2
  • Do not assume high confidence means correctness: a judge can confidently accept fluent prose containing a fabricated figure. section 10.5.2

A practical escalation rule

Route an evaluation to a stronger judge or human reviewer when the Tier 1 judge reports low confidence, produces a borderline score, varies across reruns, or reacts differently when answer order is swapped. section 10.5.2

How AI Model Evaluation covers it

How deep it goes
Explains confidence-based escalation and its blind spot, but does not define a calibration method.
Examples
Uses a borderline score of 3/5, rerun variance, swapped answer order, and a fabricated figure as examples.
What it assumes
Assumes familiarity with judge tiers, verdict scores, repeated sampling, and human review.
What you can do after
Operate escalation by routing uncertain cases upward while recognizing confidently wrong cases can bypass review.
How it connects
Connects confidence with escalation, stability testing, position sensitivity, and human review.

Also explained in

Compare how 16 books explain confidence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.