liveBook · Manning

Bias detection

As explained in Machines that Think

Bias testing is deliberate evaluation of whether a system treats equivalent or comparable cases differently without task-justified reasons.

Bias testing probes whether a system treats equivalent or comparable cases differently without a task-justified reason. It shifts attention from why one output occurred to whether differences recur across groups, languages, contexts, or use cases. section 7.4.3

Common approaches look for different kinds of disparities:

  • Counterfactual testing keeps most of an input unchanged while varying one attribute, then checks whether the output changes without justification. section 7.4.3
  • Subgroup and intersectional evaluation compares results across groups or contexts, including combinations of attributes that aggregate scores can obscure. section 7.4.3
  • Benchmarks make evaluation repeatable for known risks, but passing one does not establish fair behavior in other workflows. section 7.4.3

A controlled comparison

Compare resumes with the same qualifications but different names. An output difference that the task cannot justify is a signal for further investigation, not by itself proof of the system’s underlying mechanism. section 7.4.3

Figure 7.7 Techniques for exposing bias. Bias testing compares original and modified inputs to check whether small changes produce unjustified differences in output. Subgroup evaluation compares outcomes across relevant groups or contexts to detect disparities hidden by average performance. Benchmark evaluation uses standardized test sets to measure known failure patterns, while still requiring caution because benchmarks only reveal the behaviors they are designed to test.

The figure contrasts controlled input changes, subgroup comparisons, and standardized benchmarks, while noting that benchmarks cover only the behaviors they test. section 7.4.3

Structured tests can miss unusual wording, ambiguous context, multi-turn pressure, or unanticipated combinations of details. Bias testing therefore contributes evidence to an audit; it does not guarantee fair behavior across every use. section 7.4.3 section 7.5

How Machines that Think covers it

How deep it goes
Compares controlled input changes and group-level results, but warns that benchmarks reveal only tested behaviors and passing does not establish fairness elsewhere.
Examples
Compares resumes with different names, prompts about different groups, and profiles changed only by location or school.
What you can do after
Evaluate whether comparable cases receive unjustified differences by varying one attribute or examining outcomes across groups.
How it connects
Connects counterfactual testing with subgroup and intersectional evaluation, benchmark evaluation, auditing, red teaming, and ongoing monitoring.

Also explained in

Compare how 3 books explain bias detection →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.