Automation bias is the tendency to accept an automated recommendation or default choice because evaluating it independently takes more effort. In an LLM system, fluent answers can make occasional errors especially difficult to notice. section 8.2 section 8.0
The risk grows when a system usually works well: people stop checking carefully and may accept the rare wrong answer. If suggestions are wrong often enough to preserve vigilance, however, the automation may slow work more than it helps. section 8.2
Practical ways to reduce automation-bias risk include:
- Add an escape path to a human representative when the conversation becomes novel, too long, or otherwise unsuitable for the LLM. section 8.2
- Change the process so a person gives final approval, rather than requiring constant vigilance throughout an automated conversation. section 8.2.1
- Limit automation, require authentication or collateral, and vary human review when users might exploit predictable controls. section 8.2.1
- Use retrieval augmented generation to ground answers in known information, while recognizing that better accuracy does not remove automation bias. section 8.2
Tech-support chatbot
A chatbot retrieves relevant manuals and proposes an answer for a technician. A human reviewer may approve it automatically because the answer usually looks right. A safer design can route unusual cases to a higher support tier or require explicit final approval for every outcome. section 8.2 section 8.2.1
Figure 8.2 A naive approach toward implementing a “human in the loop” system that uses an LLM paired with a database of relevant information to produce output that is ultimately reviewed by and possibly corrected by a human worker
Trace the naive human-in-the-loop pipeline: an LLM uses a database of relevant information to generate an answer that a human reviews and possibly corrects. section 8.2
