Static content-based routing sends examples to a stronger evaluator according to input properties, not the lower-tier judge’s confidence or verdict. It protects against correlated blind spots that threshold tuning cannot detect. section 10.5.2
Useful routing rules include:
- Escalate high-severity categories, responses containing numeric claims or citations, unusually long or sparse source context, or traffic from an unvalidated user segment. section 10.5.2
- Send a fixed random percentage of high-confidence lower-tier verdicts to a stronger evaluator or human reviewer as a blind audit sample. section 10.5.2
- Track reviewer disagreement as a confident-error rate; increases can reveal competence drift on a traffic slice before aggregate scores change. section 10.5.2
A practical evaluation pipeline
Route a response with numeric claims to Tier 2 regardless of Tier 1’s confidence. Separately, sample some high-confidence Tier 1 verdicts for Tier 2 or human review, then compare outcomes. section 10.5.2
