Tiered judging controls evaluation cost and latency by applying a lightweight judge broadly, then reserving stronger, more expensive adjudication for examples that need it. It treats judging as an engineering system with an intentional evaluation budget rather than maximizing judge sophistication on every example. section 10.5 section 10.5.1
A practical tiered pipeline typically follows this anatomy:
- Tier 1 evaluates every example with a fast, low-cost model using a simple rubric or checklist. section 10.5.1
- Tier 2 re-evaluates examples that Tier 1 cannot confidently resolve, using a stronger judge and potentially reasoning-first, panel, or argue-then-judge methods. section 10.5.1
- Verify the routing assumption offline: run both tiers on the same set and check whether Tier 1 accuracy rises with its reported confidence. section 10.5.1
- Keep escalation and routing logic in pipeline code, and pair confidence triggers with at least one trigger that does not depend on self-confidence. section 10.7
Offline routing check
Run Tier 1 and Tier 2 across the same offline evaluation set. Group Tier 1 verdicts by self-reported confidence and compare their accuracy with Tier 2 outcomes. If confidence does not predict correctness, do not rely on confidence alone to select escalations. section 10.5.1
The efficiency gain depends on the task distribution: most outputs must be easy calls, while only a minority lie near the decision boundary. That assumption requires verification, because tiering does not guarantee that a cheap judge handles clear cases correctly. section 10.5.1
Abstention or deferral provides another safety valve when the judge lacks sufficient context. A rising defer rate usually points to a data-pipeline problem rather than a judging problem. section 10.7
