liveBook · Manning

Representational harm

As explained in Machines that Think

Representational harm is damage caused when language or information technologies systematically misrepresent, erase, or distort groups and experiences.

Representational harm occurs when language technologies systematically underrepresent, erase, stereotype, or misdescribe groups and experiences. Bias can arise not only from skewed associations, but also from missing perspectives: what is rarely recorded leaves little statistical trace. section 7.3.4

When training data sparsely represents a language or cultural context, a model has fewer detailed examples to draw on. Its responses may flatten distinctions, omit locally grounded knowledge, or fall back on more dominant narratives. Historical training data can also leave later events and developments outside the model’s learned patterns. section 7.3.4

Representational harm differs from allocative harm, which concerns how systems distribute opportunities, resources, or services. The harms can interact: misrepresentation or uneven recognition may shape decisions about interviews, healthcare, education, or institutional support. section 7.3.5

Examples show how gaps in representation can produce unequal system behavior:

  • A healthcare algorithm underestimated Black patients’ needs when it used healthcare costs as a proxy for medical need. section 7.3.5
  • Commercial speech-recognition systems had higher word error rates for Black speakers, while gender-classification systems performed worse on darker-skinned women. section 7.3.5
  • Hate-speech detection could mistake African American English for toxicity when data and annotation did not account for dialect. section 7.3.5

How Machines that Think covers it

How deep it goes
Traces how sparse data leaves weak statistical traces, prompting generalized or dominant-culture approximations, and links these failures to uneven treatment.
Examples
Uses healthcare-need scoring, speech recognition, gender classification, and hate-speech detection to illustrate unequal representation and treatment.
What you can do after
Evaluate how underrepresentation can distort outputs and distinguish representational harm from allocation of opportunities, resources, or services.
How it connects
Connects bias, training-data representation, language-model outputs, and allocative harm.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.