Fairness evaluates whether a model serves relevant user or content groups equitably rather than hiding disparities behind a healthy overall metric. Treat fairness scores as product diagnostics: document their assumptions, connect measures such as demographic parity to user experience, and validate their relevance against user feedback, A/B tests, or retrospective analyses. section 3.5
A practical fairness evaluation loop includes:
- Include underrepresented groups, rare behaviors, and extreme conditions in diagnostic scenarios instead of evaluating only the most common experience. section 3.5
- Inspect trade-offs explicitly: correcting disparities through feature reweighting may temporarily reduce precision, requiring product-level discussion rather than metric-only optimization. section 3.5
- Stratify online experiments by language, geography, and tenure; apply subgroup-specific non-inferiority margins instead of relying on the global average. section 8.4.1
- Log cohort attributes so monitoring can slice outcomes and enforce those subgroup guardrails after deployment. section 8.4.1
Movie recommendations across user cohorts
A recommender may perform well for long-tenured users while under-serving new users or minority-language speakers. Offline cohort analysis should also test niche preferences, such as obscure independent films, alongside mainstream tastes. Bring these cohorts into online monitoring; otherwise, an acceptable A/B-test average can conceal widening disparities. section 3.5 section 8.4.1













