liveBook · Manning

Tiered judging

As explained in AI Model Evaluation

Tiered judging is an evaluation technique that applies a lower-cost judge broadly and routes selected examples to a stronger judge for higher-fidelity adjudication.

Tiered judging controls evaluation cost and latency by applying a lightweight judge broadly, then reserving stronger, more expensive adjudication for examples that need it. It treats judging as an engineering system with an intentional evaluation budget rather than maximizing judge sophistication on every example. section 10.5 section 10.5.1

A practical tiered pipeline typically follows this anatomy:

  • Tier 1 evaluates every example with a fast, low-cost model using a simple rubric or checklist. section 10.5.1
  • Tier 2 re-evaluates examples that Tier 1 cannot confidently resolve, using a stronger judge and potentially reasoning-first, panel, or argue-then-judge methods. section 10.5.1
  • Verify the routing assumption offline: run both tiers on the same set and check whether Tier 1 accuracy rises with its reported confidence. section 10.5.1
  • Keep escalation and routing logic in pipeline code, and pair confidence triggers with at least one trigger that does not depend on self-confidence. section 10.7

Offline routing check

Run Tier 1 and Tier 2 across the same offline evaluation set. Group Tier 1 verdicts by self-reported confidence and compare their accuracy with Tier 2 outcomes. If confidence does not predict correctness, do not rely on confidence alone to select escalations. section 10.5.1

The efficiency gain depends on the task distribution: most outputs must be easy calls, while only a minority lie near the decision boundary. That assumption requires verification, because tiering does not guarantee that a cheap judge handles clear cases correctly. section 10.5.1

Abstention or deferral provides another safety valve when the judge lacks sufficient context. A rising defer rate usually points to a data-pipeline problem rather than a judging problem. section 10.7

How AI Model Evaluation covers it

How deep it goes
Explains two tiers, routing assumptions, confidence validation, and pipeline placement, without prescribing universal models or thresholds.
Examples
Uses a two-tier evaluator and an offline set comparing Tier 1 confidence with Tier 2 outcomes.
What it assumes
Requires an evaluation pipeline and judge-confidence signal, while verifying rather than assuming that confidence predicts correctness.
What you can do after
Build it from scratch by separating first-pass judging, escalation logic, stronger adjudication, and offline routing validation.
How it connects
Connects LLM-as-a-judge with escalation, confidence-based routing, abstention, validation, and evaluation budget management.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.