liveBook · Manning

Non-inferiority margin

As explained in AI Model Evaluation

A predefined maximum acceptable difference from a reference outcome that must not be exceeded to conclude that performance is not meaningfully worse.

A non-inferiority margin is a predefined maximum acceptable deterioration from a reference outcome. It turns product trade-offs into an explicit evaluation rule: a treatment may improve its primary metric only if protected outcomes remain within the allowed loss. section 8.1.2

Use the margin to make evaluation contracts specific and actionable:

  • Pair a primary success metric with guardrails such as retention, satisfaction, completion rate, skips, or thumbs-downs. section 8.1.2
  • Apply subgroup-specific margins when global averages could hide performance gaps across language, geography, or tenure. section 8.4.1
  • Trigger alerts only when a statistically confident change exceeds a meaningful bound, reducing noise and alert fatigue. section 8.7.3

Movie recommendation experiment

A recommender may target higher click-through rate while requiring consumption not to fall more than 2% versus control. The treatment must show superiority on click-through rate and non-inferiority on the consumption guardrail. The same protection can be applied to each monitored user cohort. section 8.1.2 section 8.4.1

Margins prevent narrow proxy gains from masquerading as product improvement. They help expose reward hacking, protect trust and retention, and constrain short-term feedback effects that could damage long-term usefulness. section 8.3.1 section 8.2.1

How AI Model Evaluation covers it

How deep it goes
Explains definition, metric pairing, subgroup application, and statistically confident alerting, but does not derive the statistical test.
Examples
Uses a movie recommender, a 2% consumption bound, cohort monitoring, and retention alerts as concrete applications.
What it assumes
Assumes familiarity with online metrics, A/B testing, guardrails, and user cohorts.
What you can do after
Evaluate/understand model changes by setting primary metrics, guardrails, subgroup margins, and actionable violation thresholds.
How it connects
Connects online metrics with guardrail metrics, reward hacking, fairness, feedback loops, and alert fatigue.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.