liveBook · Manning

Fairness

As explained in AI Model Evaluation

Fairness is the extent to which a model treats relevant user or content groups equitably and without unjustified disparities.

Fairness evaluates whether a model serves relevant user or content groups equitably rather than hiding disparities behind a healthy overall metric. Treat fairness scores as product diagnostics: document their assumptions, connect measures such as demographic parity to user experience, and validate their relevance against user feedback, A/B tests, or retrospective analyses. section 3.5

A practical fairness evaluation loop includes:

  • Include underrepresented groups, rare behaviors, and extreme conditions in diagnostic scenarios instead of evaluating only the most common experience. section 3.5
  • Inspect trade-offs explicitly: correcting disparities through feature reweighting may temporarily reduce precision, requiring product-level discussion rather than metric-only optimization. section 3.5
  • Stratify online experiments by language, geography, and tenure; apply subgroup-specific non-inferiority margins instead of relying on the global average. section 8.4.1
  • Log cohort attributes so monitoring can slice outcomes and enforce those subgroup guardrails after deployment. section 8.4.1

Movie recommendations across user cohorts

A recommender may perform well for long-tenured users while under-serving new users or minority-language speakers. Offline cohort analysis should also test niche preferences, such as obscure independent films, alongside mainstream tastes. Bring these cohorts into online monitoring; otherwise, an acceptable A/B-test average can conceal widening disparities. section 3.5 section 8.4.1

How AI Model Evaluation covers it

How deep it goes
Explains metric trust, diverse diagnostic scenarios, precision trade-offs, cohort stratification, event logging, and subgroup guardrails.
Examples
Uses movie recommendations for niche versus mainstream tastes, new users, and minority-language speakers.
What you can do after
Apply it via cohort-stratified experiments, subgroup non-inferiority margins, attribute logging, and fairness diagnostics.
How it connects
Connects fairness to offline evaluation, demographic parity, A/B testing, cohort analysis, and online monitoring.

Also explained in

Compare how 14 books explain fairness →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.