liveBook · Manning

Evaluation escalation

As explained in AI Model Evaluation

The routing of an evaluation example from a cheaper evaluator to a stronger evaluator or human reviewer when specified conditions are met.

Evaluation escalation routes an example from a cheap Tier 1 judge to a stronger judge or human reviewer instead of accepting the first verdict. It concentrates expensive review on cases that signal uncertainty, but cannot reliably catch confidently wrong judgments. section 10.5.2

Common escalation triggers include:

  • A borderline score, such as 3 on a five-point scale when 3 separates pass from fail. section 10.5.2
  • High variance across repeated samples, indicating an unstable judgment. section 10.5.2
  • Position sensitivity revealed by a swapped-order test. section 10.5.2
  • Explicitly low confidence reported by the Tier 1 judge. section 10.5.2

The confident fabrication

A judge may accept a fabricated figure because fluent surrounding prose makes the answer appear reliable. If it remains confident and consistent, escalation thresholds never route the example upward. section 10.5.2

How AI Model Evaluation covers it

How deep it goes
Explains four triggers and the correlated-error blind spot, but does not provide a remedy for confident systematic misjudgment.
Examples
Uses a borderline score, repeated-sampling variance, swapped-order inconsistency, and a confidently accepted fabricated figure.
What it assumes
Assumes a Tier 1 judge produces verdicts, scores, repeated samples, or confidence signals for routing.
What you can do after
Operate a tiered evaluator that forwards borderline, unstable, position-sensitive, or low-confidence cases for stronger review.
How it connects
Connects LLM-as-a-judge with confidence, stability testing, position sensitivity, swapped-order testing, and human review.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.