Fairness evaluation
Fairness evaluation is the assessment of whether a model or decision process produces unjustified disparities in outcomes, treatment, or errors across relevant groups.
AI Model Evaluation
A fairness evaluation assesses whether a model's outcomes or error rates differ unfairly across protected or otherwise relevant groups.
- How it is used
- Uses group comparisons as offline diagnostic checks before experimentation or deployment.
- How deep it goes
- Defines demographic parity and equality of opportunity, shows disparity calculations, and stops short of prescribing universal fairness thresholds.
- Examples
- Uses blockbuster and long-tail movie recommendations, group disparity ratios, and unequal image-cropping exposure as examples.
- What you can do after
- Diagnose potential group disparities and decide which model behaviors require further investigation before launch.
AI Governance
Fairness testing evaluates whether an AI system produces unjustified disparities or inequitable outcomes for people or groups.
- How it is used
- Detects and investigates inequitable outcome differences in LLM-based systems, including systems that cannot be retrained.
- How deep it goes
- Details paired scenarios, outcome labels, group-rate measurement, and release controls, but leaves harm selection and fairness judgments unresolved.
- Examples
- Uses ClaimAssist claim decisions, paired sensitive-signal scenarios, group denial rates, and output checks as test examples.
- What you can do after
- Diagnose disparities by running paired tests, comparing group outcome rates, and escalating threshold breaches for review.
Also covered in
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
Related concepts
- paired testinguses · A testing technique that compares system outcomes for matched scenarios differing in a selected characteristic while other relevant conditions are held constant.
- evaluationkind of · Evaluation is the systematic assessment of a system, model, process, or outcome against relevant criteria or objectives using evidence.
- demographic parityuses · Demographic parity is a fairness criterion requiring favorable outcome or positive prediction rates to be equal across specified demographic groups.
Questions about fairness evaluation
What is fairness evaluation?
Fairness evaluation is the assessment of whether a model or decision process produces unjustified disparities in outcomes, treatment, or errors across relevant groups.
Which Manning books explain fairness evaluation?
AI Model Evaluation and AI Governance.
What should I know before learning fairness evaluation?
Manning books that teach fairness evaluation usually introduce language model first.
What is fairness evaluation a kind of?
Fairness evaluation is a kind of evaluation. Evaluation is the systematic assessment of a system, model, process, or outcome against relevant criteria or objectives using evidence.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

