liveBook · Manning

Content-based routing

As explained in AI Model Evaluation

Static content-based routing sends examples to an evaluation tier according to properties of the input rather than the lower-tier evaluator's judgment.

Static content-based routing sends examples to a stronger evaluator according to input properties, not the lower-tier judge’s confidence or verdict. It protects against correlated blind spots that threshold tuning cannot detect. section 10.5.2

Useful routing rules include:

  • Escalate high-severity categories, responses containing numeric claims or citations, unusually long or sparse source context, or traffic from an unvalidated user segment. section 10.5.2
  • Send a fixed random percentage of high-confidence lower-tier verdicts to a stronger evaluator or human reviewer as a blind audit sample. section 10.5.2
  • Track reviewer disagreement as a confident-error rate; increases can reveal competence drift on a traffic slice before aggregate scores change. section 10.5.2

A practical evaluation pipeline

Route a response with numeric claims to Tier 2 regardless of Tier 1’s confidence. Separately, sample some high-confidence Tier 1 verdicts for Tier 2 or human review, then compare outcomes. section 10.5.2

How AI Model Evaluation covers it

How deep it goes
Defines routing triggers, blind audits, and confident-error tracking, while stopping short of threshold-based detection.
Examples
Uses severity categories, numeric claims, citations, unusual context length, unvalidated segments, and random high-confidence audits.
What you can do after
Operate/configure evaluation tiers with independent routing and monitor confident-error rates for competence drift.
How it connects
Connects escalation with evaluator uncertainty, correlated errors, drift detection, checklists, and human review.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.