liveBook · Manning

Overreliance

As explained in AI Governance

Overreliance is the tendency to defer to an automated system’s recommendations without applying independent judgment or checking supporting evidence.

A claims reviewer denies a customer’s claim after an AI assistant confidently attributes negligence to the customer. The reviewer does not check the original photos; an appeal later reveals invented details and omitted context. The failure is overreliance: treating an automated recommendation as more authoritative than the evidence. section 6.3

Overreliance is closely tied to automation bias: people favor automated recommendations, especially under time pressure. It can cause errors of commission, when someone acts on a bad suggestion, and errors of omission, when someone misses a problem the system did not flag. Repeated dependence can also weaken the underlying skills needed to assess evidence and competing explanations. section 6.3

The risk appears in different deployment models: an employee may accept a candidate shortlist without reading the CVs, while a workflow may approve claims automatically from an API response. Both treat probabilistic output as fact. A factuality guardrail can help when there is source material to check, but it is not a complete safeguard against false claims about people. section 5.4.3

Build independent judgment into the workflow:

  • Assign a decision owner and document which steps can use AI assistance, require a human, or must never be automated. Configure the system to enforce those boundaries. section 6.3.1
  • Give reviewers enough time, context and authority to disagree. Label outputs as suggestions, expose uncertainty and make rejection possible; a rubber stamp is not meaningful human review. section 5.4.2 section 6.3.1
  • Train with realistic successes and failures. Have reviewers perform tasks with and without AI, identify warning signals and turn observed mistakes into checks against original evidence. section 6.3.1
  • Measure review quality through appeals, audit findings and supervisor scores. For high-stakes decisions, set minimum review times by risk tier and investigate consistently faster approvals. section 6.3.1
Figure 6.6 Three layers of defense against overreliance on AI: the user's own judgment, workflow design that surfaces uncertainty, and oversight that feeds learning back into the system.

The three-layer view connects individual judgment, workflow design and oversight that feeds learning back into the system. section 6.3

Agentic systems can act before a person sees a flawed output. Use human approval gates for actions that are high-risk, irreversible, or carry significant legal or financial consequences. For consequential decisions requiring human judgment, design the system as decision support rather than hidden automation. section 5.4.2

Oversight also depends on sustained practice: encourage reviewers to override suggestions, report near misses and question outputs. When AI supports decisions about people, false personal claims can have privacy consequences even if the errors were unintentional. section 6.8 section 5.4

How AI Governance covers it

How deep it goes
Defines automation bias and error types, then details accountability, training, interface choices and review monitoring.
Examples
Uses a disputed claims denial, candidate screening and AI-assisted programming to illustrate unchecked recommendations.
What you can do after
Apply workflow controls that support independent review and monitor for rubber-stamping.
How it connects
Connects automation bias with hallucinations, human-in-the-loop oversight, automated decision-making and agentic AI.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.