liveBook · Manning

Automation bias

As explained in AI Governance

Automation bias is the tendency to favor or trust recommendations from automated systems, potentially leading people to make errors or overlook relevant information.

Automation bias is the tendency to favor an automated recommendation over independent evidence or judgment. A fluent, confident AI response can appear authoritative, especially under time pressure or after the system has been mostly right. As obvious errors become less frequent, remaining errors may be harder to notice and less likely to be challenged. section 5.4.2 section 6.3

A discharge note accepted without verification

MediAssist drafts a patient’s discharge note and falsely attributes an asthma episode to failure to follow treatment. The physician accepts the professional-looking note without checking it. Although a human reviewed the output, the automated statement effectively decided what entered the record. section 5.4.2

Overreliance can produce errors of commission, when someone acts on a wrong recommendation, and errors of omission, when they miss something the system leaves out. In a claims workflow, a reviewer denied a customer’s claim after accepting an AI summary that invented details and omitted context; an appeal exposed the error. The same pattern can affect coding: in one controlled study, developers using an assistant were more likely to produce insecure code and rate it as secure. section 6.3

A human approval step is not meaningful oversight if the reviewer simply accepts the output. Reviewers need to exercise independent judgment against underlying evidence, particularly when generated statements about people could lead to unfair treatment. Workflow design can surface uncertainty, while oversight can feed lessons from failures back into the system. section 5.1 section 6.3

Figure 6.6 Three layers of defense against overreliance on AI: the user's own judgment, workflow design that surfaces uncertainty, and oversight that feeds learning back into the system.

The three-layer defense connects individual judgment with workflow design and oversight. section 6.3

How AI Governance covers it

How deep it goes
Explains the trust paradox and errors of commission and omission through worked scenarios and layered defenses.
Examples
Uses a fabricated medical discharge note, a denied insurance claim, and a coding-assistant security study.
What you can do after
Evaluate whether human review exercises independent judgment rather than automatically accepting AI output.
How it connects
Connects hallucinations, human-in-the-loop oversight, privacy, and automated decision-making.

Also explained in

Compare how 4 books explain automation bias →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.