liveBook · Manning

Bias

As explained in Machines that Think

A systematic distortion in data, model behavior, or decisions that produces uneven or unjustified outcomes.

Bias is a systematic distortion in data, model behavior, or decisions that can produce uneven or unjustified outcomes. It need not reflect explicit prejudice or malicious intent: historical outcomes, uneven representation, and apparently neutral proxies can all carry unequal patterns into a model. section 7.3.1

A decision model trained on past hiring, lending, policing, or healthcare records may learn patterns produced by unequal access. Once deployed, its decisions can influence who receives opportunities or attention, creating new data that reinforces the original pattern. section 7.3.1

Figure 7.3 Proxy bias in training data. A model trained on past hiring decisions may learn that candidates from School A are more likely to be hired, even when experience and test scores point in another direction. Because School A is all-boys, this apparently neutral feature can act as a proxy for gender, causing the model to reproduce a male-favoring pattern without using gender as an explicit input.

A hiring example shows how school attended can act as a proxy for gender, even when gender is not an explicit input. section 7.3.1

Language models inherit patterns from a different record: human expression. Their training data reflects what people chose to write and how they framed it, rather than a direct view of the world. Studies have found demographic differences in generated occupations and descriptions, as well as in conversational tone. section 7.3.2

Bias may be hard to identify in a single response. It can appear across rankings, representations, prompts, or repeated outputs, including responses that contain no obviously offensive language. Explanations can help inspect individual decisions, but a model’s stated reason may not faithfully describe what drove its answer. section 7.4 section 7.4.2

Auditing looks for systematic differences across comparable cases:

  • Counterfactual testing changes one attribute while keeping most of an input fixed, such as changing the name on a résumé while retaining its qualifications. section 7.4.3
  • Subgroup and intersectional evaluation compares outcomes across groups or contexts, including combinations that aggregate scores may obscure. section 7.4.3
  • Benchmark evaluation tests known risks consistently, but passing a benchmark does not establish fairness in another workflow. section 7.4.3

Mitigation can act at several stages, including data curation, training objectives, fine-tuning, and output constraints. No single layer fully eliminates inherited bias; ongoing monitoring and judgment remain important. section 7.5

In an artificial neuron, bias has a separate mathematical meaning: it is an adjustable constant added to the weighted inputs before the activation function. It shifts the combined signal and helps the neuron respond when inputs are weak; this parameter should not be confused with unfair or systematic bias. section 3.1.1

How Machines that Think covers it

How deep it goes
Explains sources, auditing practices, and mitigation points while noting that benchmarks and interventions have limits.
Examples
Uses hiring by school, résumé name changes, and demographic differences in generated occupations and conversational tone.
What you can do after
Evaluate and understand possible bias by comparing cases, inspecting subgroup outcomes, and interpreting benchmarks cautiously.
How it connects
Connects bias to proxy variables, explainability, counterfactual testing, subgroup evaluation, and benchmarking.

Also explained in

Compare how 38 books explain bias →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.