Bias testing probes whether a system treats equivalent or comparable cases differently without a task-justified reason. It shifts attention from why one output occurred to whether differences recur across groups, languages, contexts, or use cases. section 7.4.3
Common approaches look for different kinds of disparities:
- Counterfactual testing keeps most of an input unchanged while varying one attribute, then checks whether the output changes without justification. section 7.4.3
- Subgroup and intersectional evaluation compares results across groups or contexts, including combinations of attributes that aggregate scores can obscure. section 7.4.3
- Benchmarks make evaluation repeatable for known risks, but passing one does not establish fair behavior in other workflows. section 7.4.3
A controlled comparison
Compare resumes with the same qualifications but different names. An output difference that the task cannot justify is a signal for further investigation, not by itself proof of the system’s underlying mechanism. section 7.4.3
Figure 7.7 Techniques for exposing bias. Bias testing compares original and modified inputs to check whether small changes produce unjustified differences in output. Subgroup evaluation compares outcomes across relevant groups or contexts to detect disparities hidden by average performance. Benchmark evaluation uses standardized test sets to measure known failure patterns, while still requiring caution because benchmarks only reveal the behaviors they are designed to test.

The figure contrasts controlled input changes, subgroup comparisons, and standardized benchmarks, while noting that benchmarks cover only the behaviors they test. section 7.4.3
Structured tests can miss unusual wording, ambiguous context, multi-turn pressure, or unanticipated combinations of details. Bias testing therefore contributes evidence to an audit; it does not guarantee fair behavior across every use. section 7.4.3 section 7.5


