A generative AI system can produce an answer that sounds confident but contains fabricated or false information. This is a hallucination: the output presents an unsupported claim as fact. Plausible wording does not establish accuracy, and a false claim can be mixed with real details. section 1.1 section 5.4
A fabricated claims history
An insurance chatbot tells a customer that she has had two rear-end collisions and warns that another could trigger a policy review. In fact, her recent accident was her first in a decade. The chatbot has presented a fictional claims history as fact. section 5.4
False claims about identifiable people can be personal data even when inaccurate. They may require correction, and harmful processing may be unlawful regardless of intent. If such claims influence significant decisions, such as job application rejection, the consequences extend beyond an incorrect answer. section 5.4
Hallucinations can also invent legal facts or citations, or combine real personal details with defamatory claims. Overreliance magnifies the harm: a convincing answer can mislead the person who uses it. Generative AI can produce such errors quickly and at scale. section 1.1 section 5.4
Table 1.1 Why Generative AI Breaks Traditional Control Models (Continued)
|
Pre-deployment
assurance
|
Tests, code
review, static analysis, red-teaming (ethical hackers)
|
Role-specific
training, competency certification, scenario-based drills
|
Safety and
bias testing, adversarial prompt suites, red-teaming, interpretability tools
(e.g. probing)
|
|
Post-launch
controls
|
Logging,
health checks, intrusion detection, WAF, auto-rollback on error rate spike
|
Supervisors,
peer review, escalation protocols
|
Output
filters, rate limiting, live metrics on toxicity and drift, anomaly detection
on embeddings
|
|
Failure
detection & attribution
|
Clear (stack
traces, logs, input replay)
|
Mixed
(relies on human reporting or
observation)
|
Opaque
(failures can appear valid; root cause buried in model weights)
|
|
Memory
/ learning capability
|
None (unless
explicitly coded)
|
Present
(experience, judgment, learning over time)
|
Present
(training data, fine-tuning, context memory)
|
Compare pre-deployment and post-launch controls with the difficulty of detecting AI failures that appear valid. These general controls do not by themselves establish that a particular claim is true. section 1.1




































