Automation bias is the tendency to favor an automated recommendation over independent evidence or judgment. A fluent, confident AI response can appear authoritative, especially under time pressure or after the system has been mostly right. As obvious errors become less frequent, remaining errors may be harder to notice and less likely to be challenged. section 5.4.2 section 6.3
A discharge note accepted without verification
MediAssist drafts a patient’s discharge note and falsely attributes an asthma episode to failure to follow treatment. The physician accepts the professional-looking note without checking it. Although a human reviewed the output, the automated statement effectively decided what entered the record. section 5.4.2
Overreliance can produce errors of commission, when someone acts on a wrong recommendation, and errors of omission, when they miss something the system leaves out. In a claims workflow, a reviewer denied a customer’s claim after accepting an AI summary that invented details and omitted context; an appeal exposed the error. The same pattern can affect coding: in one controlled study, developers using an assistant were more likely to produce insecure code and rate it as secure. section 6.3
A human approval step is not meaningful oversight if the reviewer simply accepts the output. Reviewers need to exercise independent judgment against underlying evidence, particularly when generated statements about people could lead to unfair treatment. Workflow design can surface uncertainty, while oversight can feed lessons from failures back into the system. section 5.1 section 6.3
Figure 6.6 Three layers of defense against overreliance on AI: the user's own judgment, workflow design that surfaces uncertainty, and oversight that feeds learning back into the system.

The three-layer defense connects individual judgment with workflow design and oversight. section 6.3



