liveBook · Manning

Blind audit sample

As explained in AI Model Evaluation

A blind audit sample is a fixed random sample of lower-tier judgments routed independently to a stronger evaluator or human reviewer.

A blind audit sample routes a fixed random share of high-confidence Tier 1 judgments to Tier 2 or a human reviewer without relying on Tier 1's uncertainty. It exposes confident errors that uncertainty-based escalation systematically misses. section 10.5.2

Implement the audit path as a separate, predictable measurement channel:

  • Select a fixed random percentage of Tier 1's high-confidence verdicts. section 10.5.2
  • Send those cases to Tier 2 or a human reviewer independently of Tier 1's confidence. section 10.5.2
  • Track Tier 1–reviewer disagreements as a separate confident-error rate. section 10.5.2
  • Treat a rising rate as an early signal that the cheaper judge has drifted beyond its competence on a traffic slice. section 10.5.2

How AI Model Evaluation covers it

How deep it goes
Defines routing, disagreement tracking, and drift interpretation, while stopping short of specifying a sampling percentage.
Examples
Uses high-confidence verdicts randomly reassessed by Tier 2 or a human reviewer.
What it assumes
Assumes a tiered evaluation pipeline with a cheaper Tier 1 judge and stronger Tier 2 or human review.
What you can do after
Diagnose judge drift on traffic slices before aggregate quality metrics visibly deteriorate.
How it connects
Connects escalation with Tier 1/Tier 2 evaluation, model drift, and confident-error rates.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.