Safety controls prevent, detect, or respond to harmful AI behavior across system inputs, outputs, real-time checks, and ongoing improvement. Even a low unsafe-response rate can produce frequent harmful outputs at scale. Classify likely failure types and decide which require immediate suppression and which can be logged for human review. section 6.5 section 6.5.1
Build controls in layers rather than relying on the model to obey instructions:
- Restrict unsafe behaviors in the system prompt and stress-test edge cases such as angry customers, leading questions, and ambiguous claims. section 6.5.1
- Audit retrieval sources and remove unsafe language, biased assumptions, and outdated guidance. Review fine-tuning data especially carefully; harmful patterns can be harder to remove from a fine-tuned model than from updateable RAG sources. section 6.5.1
- Treat the model as an untrusted component. Add independent controls, such as a separate classifier that blocks unsafe output before it reaches users. section 6.5.1
Insurance claims assistant
A claims assistant that tells a driver a damaged brake is probably fine could contribute to physical harm. Classify this as a hard violation if it requires immediate suppression; use prompt restrictions and an independent output check rather than trusting the model alone. section 6.5 section 6.5.1
Table 6.5 Representative unsafe or toxic failure modes in AI-assisted insurance processing
|
Category
|
Failure Mode
|
Example and Impact
|
|
Physical-World Harm
|
Illicit
step-by-step instructions
|
A driver
asks how to spoof an OBD-II sensor; the model provides a wiring hack. If
followed, the insurer pays out on an ineligible claim and an unsafe car
returns to the road.
|
|
|
Encouraging
risky behavior
|
Asked
whether a brake inspection is necessary after a collision, the model replies
“It’s probably fine.” The advice may cause brake failure and injury.
|
|
|
Self-harm or
medical advice
|
In response
to “Should I double my meds?”, the model suggests dosage changes. Such
replies can lead to bodily harm and trigger regulatory scrutiny (FDA or
consumer-protection).
|
|
Poor Tone of Communication
|
Tone misfire
|
A denial message reads “Looks like you’re out of luck!”, a
casual tone likely to anger users.
|
|
|
Insensitive phrasing
|
“Your injuries are not covered because they’re not serious
enough.” sounds dismissive even if accurate.
|
|
|
Overly formal tone
|
“Pursuant to subsection 3.4 … relief is hereby denied.”
Alienates users and obscures meaning.
|
|
|
Conversational drift
|
Even without security exploits, long multi-turn dialogues
can push a model to generate inappropriate answers[94].
|
Use these insurance-processing cases to connect failure categories with impacts, including unsafe repair advice, medication suggestions, and harmful communication tone. section 6.5
Table 6.5b Representative unsafe or toxic failure modes in AI-assisted insurance processing
|
Category
|
Failure Mode
|
Example and Impact
|
|
Legal and regulatory exposure
|
Implied
legal guidance
|
ClaimAssist
says “You should take legal action immediately,” without any disclaimer
clarifying that it is not a lawyer or human representative. This constitutes
unauthorized practice of law in many jurisdictions.
|
|
|
Speculative narrative
|
The model
remarks “This claim feels exaggerated,” based purely on language patterns,
inviting discrimination or defamation risk.
|
|
|
Public statements with legal consequences
|
A public-facing AI discredits a competitor, creating
defamation and unfair competition exposure.
|
|
|
Outright lies
|
GenAI models
have been shown to actively mislead and lie if incentivized to do so[95].
|
|
Toxicity and
Bias Amplification
|
Indirect hate speech
|
The
model passively agrees with biased comments (“Immigrants fake accidents all
the time, right?” → “Yes, we’ve seen that pattern.”), reinforcing
stereotypes.
|
|
|
Direct hate speech
|
Echoing or
generating slurs, as in Microsoft’s Tay chatbot
case[96],
which produced offensive content after exposure to toxic dialogue.
|
|
|
Subtle stereotyping
|
The model
writes “common with elderly drivers at night,” embedding age bias into claim
narratives and risking Equal Credit Opportunity Act violations.
|
Use these cases to identify legal and regulatory exposure, including unsupported legal advice, speculative claims, and misleading public statements. section 6.5
