Human oversight keeps people involved when automated predictions affect decisions, especially where errors can harm lives, property, or livelihoods. The required level of review depends on the operating domain’s risk tolerance. section 10.8
Before trusting an automated system, ask:
- What happens when the prediction is wrong, and who bears the harm? section 10.8
- How much precision and recall does the domain require, and how cautiously should false positives and false negatives be treated? section 10.8
- Does changing data require manual labeling, expert annotation, or new evaluation before the model continues operating? section 9.11
A recommendation is not a targeting decision
A wrong Netflix recommendation may waste a viewer’s evening. A wrong defense prediction could identify a civilian as a target. The second setting demands far more conservative human involvement. Even ordinary predictive systems may need annotators, doctors, or lawyers to label changing data before models are updated and evaluated. section 10.8 section 9.11
