A claims reviewer denies a customer’s claim after an AI assistant confidently attributes negligence to the customer. The reviewer does not check the original photos; an appeal later reveals invented details and omitted context. The failure is overreliance: treating an automated recommendation as more authoritative than the evidence. section 6.3
Overreliance is closely tied to automation bias: people favor automated recommendations, especially under time pressure. It can cause errors of commission, when someone acts on a bad suggestion, and errors of omission, when someone misses a problem the system did not flag. Repeated dependence can also weaken the underlying skills needed to assess evidence and competing explanations. section 6.3
The risk appears in different deployment models: an employee may accept a candidate shortlist without reading the CVs, while a workflow may approve claims automatically from an API response. Both treat probabilistic output as fact. A factuality guardrail can help when there is source material to check, but it is not a complete safeguard against false claims about people. section 5.4.3
Build independent judgment into the workflow:
- Assign a decision owner and document which steps can use AI assistance, require a human, or must never be automated. Configure the system to enforce those boundaries. section 6.3.1
- Give reviewers enough time, context and authority to disagree. Label outputs as suggestions, expose uncertainty and make rejection possible; a rubber stamp is not meaningful human review. section 5.4.2 section 6.3.1
- Train with realistic successes and failures. Have reviewers perform tasks with and without AI, identify warning signals and turn observed mistakes into checks against original evidence. section 6.3.1
- Measure review quality through appeals, audit findings and supervisor scores. For high-stakes decisions, set minimum review times by risk tier and investigate consistently faster approvals. section 6.3.1
Figure 6.6 Three layers of defense against overreliance on AI: the user's own judgment, workflow design that surfaces uncertainty, and oversight that feeds learning back into the system.

The three-layer view connects individual judgment, workflow design and oversight that feeds learning back into the system. section 6.3
Agentic systems can act before a person sees a flawed output. Use human approval gates for actions that are high-risk, irreversible, or carry significant legal or financial consequences. For consequential decisions requiring human judgment, design the system as decision support rather than hidden automation. section 5.4.2
Oversight also depends on sustained practice: encourage reviewers to override suggestions, report near misses and question outputs. When AI supports decisions about people, false personal claims can have privacy consequences even if the errors were unintentional. section 6.8 section 5.4
