An applicant-screening system gives lower scores to candidates from certain ethnic backgrounds than to comparably qualified applicants. It does not directly use ethnicity, but it relies heavily on educational background, which correlates with racial and socioeconomic factors. Bias can therefore persist even when an obvious discriminatory input is excluded. The resulting decisions put applicants at a disadvantage and create legal, ethical, and reputational risks. section 11.3
Trace skewed outcomes to three possible sources:
- Training data bias: Historical decisions or unrepresentative data can favor some groups. A hiring dataset dominated by applicants from elite universities may disadvantage equally qualified candidates with less traditional backgrounds. A language model can likewise underrepresent demographics when its training data has gaps. section 11.3.1 section 5.1.1
- Algorithmic bias: A model can amplify patterns even when its data is balanced. For example, clustering applicants by geography or gender may produce recommendations that overlook more relevant qualifications. section 11.3.2
- Feedback loop bias: If someone accepts a biased recommendation and that decision becomes a positive fine-tuning example, future recommendations can reinforce the same skew. section 11.3.3
Figure 11.7 Bias can originate from the training data, the AI algorithm, and the feedback loops with users.
Use the three-source diagram to check training data, the algorithm, and user feedback rather than assuming one fix will address every cause. section 11.3
Mitigation is a continuing product and governance process:
- Audit demographic representation before training, add data where groups are underrepresented, and monitor data drift as distributions change. section 11.3.1
- Examine which features influence decisions, test whether groups receive disproportionately negative outcomes before release, and repeat fairness audits as benchmarks change. section 11.3.2
- Require human review before recommendations affect consequential decisions or future training; supplement historical recommendations with fresh data and alert reviewers when scores shift unevenly. section 11.3.3
- Tell legal and compliance partners what checks exist and where limited or unbalanced data may affect predictions. Treat fairness, user trust, and applicable regulatory requirements as product concerns—not just model-performance concerns. section 12.2.4 section 11.3.4
Historical decisions carried forward
Amazon’s 2018 hiring-tool case illustrates the risk of training on past decisions: historical hiring data favored male applicants, and the tool penalized female candidates. The practical check is not merely whether a protected attribute is an explicit input, but whether the data and resulting outcomes carry forward unequal treatment. section 11.3.4 section 11.3
