liveBook · Manning

Social bias

As explained in Machines that Think

A socially contingent pattern absorbed from historical data or practices that influences an artificial system’s behavior.

Social bias in an artificial system is a socially contingent pattern inherited from historical data or practices that influences its behavior. Records contain stereotypes, omissions, institutional decisions, and uneven representation; a system can absorb these alongside useful predictive signals. Biased behavior need not imply explicit prejudice or human-like moral judgment. section 7.3 section 7.0

In language models, repetition can turn a social association into predictive weight. If texts repeatedly pair nurses with women and engineers with men, those pairings can become more probable continuations. The model does not assess whether they are fair or representative. Learned associations can also be recombined and appear in contexts where the original examples were sparse or indirect. section 7.3.3

Figure 7.4 Repetition turns association into predictive weight. When certain words, roles, or descriptions appear together frequently in training data, language models learn those pairings as stronger statistical patterns. During generation, the model may default to the dominant association because it has become the more probable continuation, even when other alternatives exist.

The figure traces how repeated associations become stronger predictive patterns during generation. section 7.3.3

The consequences extend beyond wording. Representational harm occurs when a system stereotypes, erases, or misdescribes people. Allocative harm occurs when scores, rankings, summaries, or recommendations affect access to opportunities, resources, or services. section 7.3.5

A proxy for medical need

A commercial healthcare algorithm systematically underestimated Black patients’ needs because it used healthcare cost as a proxy for medical need. Historically lower spending on Black patients with the same level of illness was interpreted as lower need. section 7.3.5

Uneven performance can also produce uneven treatment. Commercial speech-recognition systems have shown higher word error rates for Black speakers than for white speakers; errors can matter when transcripts feed medical records, interviews, education, or legal processes. Aggregate performance can likewise conceal severe differences between groups. section 7.3.5

Explainability can help expose questionable signals. Feature attribution estimates which inputs contributed to a prediction, while counterfactual explanations ask what change would have altered an outcome. For example, inspecting a loan decision may reveal reliance on proxies for socioeconomic background rather than relevant financial factors. section 7.4.1

Mitigation can act at several stages: curating and documenting data, changing training objectives or representations, fine-tuning behavior, supplying curated context, and constraining outputs. These are layers of correction, not a single cure. Data cannot become a neutral mirror of reality simply through curation, and alignment reinforces learned tendencies rather than guaranteed rules. section 7.4.4

Figure 7.8 Bias mitigation across the model lifecycle. Bias can be addressed at several points: curating and documenting training data before learning begins, changing training objectives or internal representations during model development, using fine-tuning and alignment to reinforce desired behavior, grounding generation in curated context, and filtering or constraining outputs before they reach the user. These layers can reduce harmful behavior, but none removes the need to keep inspecting how bias enters and reappears in the system.

The figure maps mitigation options across the model lifecycle and emphasizes continued inspection. section 7.4.4

How Machines that Think covers it

How deep it goes
Explains how repeated associations acquire predictive weight and surveys mitigation layers without promising a cure.
Examples
Uses healthcare cost as a proxy for need and disparities in speech recognition to illustrate uneven outcomes.
What you can do after
Evaluate and understand how inherited patterns can shape system outputs and unequal outcomes, and where mitigation can intervene.
How it connects
Connects social bias to alignment, proxy variables, explainable AI, and representational and allocative harm.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.