A hallucination is an AI output that presents fabricated or unsupported information as reliable. It is a particular trust risk in generative AI, which can confidently state something it has made up. Direct user interaction increases the exposure: requests and responses can vary widely, and the product team cannot control every input or output. section 10.2.2 section 5.2.1
An order that the data does not support
An AI recommends ordering a product for a market with no demand after misinterpreting a correlation. A low-confidence flag gives a planner a reason to check the recommendation against the data before acting. The concern is not simply that the output is wrong, but that it could be treated as a sound basis for a business decision. section 12.3.2
Handle hallucinations according to how people use the output:
- Identify what an unsupported claim would look like in the specific workflow, rather than treating every AI error alike. section 10.2.2
- Ask whether users can spot it. An output that contradicts their knowledge may be obvious; an error buried in an aggregate may not be. section 10.2.2
- Set tolerance by stakes. A fanciful brainstorming suggestion may spark a useful idea, while an error in a medical context is far less acceptable. section 10.2.2
- Help users verify before acting. Explain warning signals such as low-confidence flags, communicate when errors are more likely, and monitor performance and feedback. section 12.3.2
Model selection can reduce risk, not remove the need for judgment: strong world knowledge is one criterion for reducing hallucinations in a model exposed directly to users. The product experience must also make uncertainty understandable so people can decide when to check an answer. section 5.2.1 section 12.3.2
