A non-inferiority margin is a predefined maximum acceptable deterioration from a reference outcome. It turns product trade-offs into an explicit evaluation rule: a treatment may improve its primary metric only if protected outcomes remain within the allowed loss. section 8.1.2
Use the margin to make evaluation contracts specific and actionable:
- Pair a primary success metric with guardrails such as retention, satisfaction, completion rate, skips, or thumbs-downs. section 8.1.2
- Apply subgroup-specific margins when global averages could hide performance gaps across language, geography, or tenure. section 8.4.1
- Trigger alerts only when a statistically confident change exceeds a meaningful bound, reducing noise and alert fatigue. section 8.7.3
Movie recommendation experiment
A recommender may target higher click-through rate while requiring consumption not to fall more than 2% versus control. The treatment must show superiority on click-through rate and non-inferiority on the consumption guardrail. The same protection can be applied to each monitored user cohort. section 8.1.2 section 8.4.1
Margins prevent narrow proxy gains from masquerading as product improvement. They help expose reward hacking, protect trust and retention, and constrain short-term feedback effects that could damage long-term usefulness. section 8.3.1 section 8.2.1
