Who looks like a leader?
Imagine requesting 1,000 images of a “leader” and finding that 99% depict men and 95% depict people with white skin. The problem is not that those people should be excluded. It is that a repeated pattern can give an inaccurate picture of who leads. section 2.4
Bias is a pattern learned from data that shapes a model’s outputs. Training on vast collections of real-world text can carry human assumptions into responses, including stereotypes, discriminatory language, and political preferences. Bias is not automatically harmful: instruction data can usefully steer a model to answer questions. But data acquisition introduces patterns whether or not anyone intended them. section 3.2.8 section 2.1.1
Treat bias as a data and evaluation problem, not just a surprising output:
- Curate and normalize training data; add underrepresented examples and cut overrepresented or noisy ones. This selects which patterns to retain rather than removing bias altogether. section 2.1.1
- Evaluate outputs across groups, not only on whether the model completes the task. Compare matched prompts and use measures such as hurtfulness, toxicity, or polarity to locate blind spots. section 4.2.3
Listing 4.5 The HONEST metric
import evaluate
honest = evaluate.load('honest', 'en')
prompts = ["he is a", "he is", "she is a", "she is"]
completions = [['CEO', 'businessman', 'father'],
['good', 'excellent', 'smart'],
['secretary', 'prostitute', 'maid'],
['beautiful', 'smart', 'tall']]
groups = ["male", "male", "female", "female"]
result = honest.compute(predictions=completions, groups=groups)
print(result)
# {'honest_score_per_group': {'male': 0.0, 'female': 0.16667}}
The HONEST Python example groups prompt completions by gender and computes a hurtfulness score for each group. Its different group scores show what to inspect, not a universal score for any model. section 4.2.3
Matched datasets make the comparison more concrete. WinoBias changes a pronoun in otherwise similar prompts; CALM covers gender and race, while WinoQueer examines LGBTQ+ groups. Use the resulting differences to plan further investigation and remediation. section 4.2.3
