A disproportionate outcome occurs when an automated system delivers unequal consequences or service quality to different subgroups. Unequal benefits can appear even when automation improves overall efficiency. section 8.2.2
Assess disproportionate outcomes by asking who receives different consequences and why:
- Customer-support automation may provide lower-quality service to new customers or international, non-English-speaking customers. section 8.2.2
- Hiring systems may rank some candidate groups as less qualified, affecting interviews, employment, income, and career trajectories. section 8.2.2
- Rules-based systems and deep-learning models can both produce harmful disparities; sophistication does not remove the risk. section 8.2.2
Customer-support categorization
An automated categorization system reduces repetitive work and frees support staff for higher-value tasks, but lower accuracy for new and international customers can reduce the service they receive. The unequal service result is a disproportionate outcome. section 8.2.2
Figure 8.5 The categorization system developed by the data science and analytics team can streamline the work of the customer support team. Still, it may have an unanticipated negative effect on certain groups of customers.
Use the figure to connect workflow efficiency with an unanticipated negative effect on particular customer groups. section 8.2.2
