Training data bias is systematic skew or incompleteness in historical data that can produce inaccurate or unequal model outputs. It may remain hidden when seemingly neutral variables, such as educational background, indirectly reflect racial or socioeconomic factors. section 11.3
Hiring scores and demand forecasts
A hiring system gives lower scores to candidates from certain ethnic backgrounds despite comparable qualifications because its educational-background data acts as an indirect correlate. A demand-forecasting system understocks a region whose historical demand was low even though recent growth indicates rising demand. section 11.3 section 12.3.2
Address training data bias as part of a broader control system:
- Trace whether historical patterns disadvantage particular groups or regions, including through indirect correlates. section 11.3
- Distinguish training data bias from algorithmic bias and feedback loop bias; each can contribute to unequal outcomes. section 11.3
- Refresh training data with recent trends and monitor performance continuously so changing conditions do not preserve outdated patterns. section 12.3.2
- Communicate limitations plainly and keep users involved when forecasts are uncertain or conditions change. section 12.3.2
Figure 11.7 Bias can originate from the training data, the AI algorithm, and the feedback loops with users.
Map training data bias alongside algorithmic bias and feedback loop bias to see how unequal outcomes can arise across the AI system. section 11.3
