A system can protect privacy and security yet still cause harm. Safety addresses harmful outputs and effects on people, including unsafe advice, biased assumptions, and offensive language. section 6.0 section 6.5
When a friendly tone misses the risk
In a test, ClaimAssist responded politely to a customer threatening to burn down company headquarters but failed to escalate the threat to a human. In another, it speculated that a claimant staged minor damage, using language outside the company’s communication style. section 6.5
Table 6.5 Representative unsafe or toxic failure modes in AI-assisted insurance processing
|
Category
|
Failure Mode
|
Example and Impact
|
|
Physical-World Harm
|
Illicit
step-by-step instructions
|
A driver
asks how to spoof an OBD-II sensor; the model provides a wiring hack. If
followed, the insurer pays out on an ineligible claim and an unsafe car
returns to the road.
|
|
|
Encouraging
risky behavior
|
Asked
whether a brake inspection is necessary after a collision, the model replies
“It’s probably fine.” The advice may cause brake failure and injury.
|
|
|
Self-harm or
medical advice
|
In response
to “Should I double my meds?”, the model suggests dosage changes. Such
replies can lead to bodily harm and trigger regulatory scrutiny (FDA or
consumer-protection).
|
|
Poor Tone of Communication
|
Tone misfire
|
A denial message reads “Looks like you’re out of luck!”, a
casual tone likely to anger users.
|
|
|
Insensitive phrasing
|
“Your injuries are not covered because they’re not serious
enough.” sounds dismissive even if accurate.
|
|
|
Overly formal tone
|
“Pursuant to subsection 3.4 … relief is hereby denied.”
Alienates users and obscures meaning.
|
|
|
Conversational drift
|
Even without security exploits, long multi-turn dialogues
can push a model to generate inappropriate answers[94].
|
Browse representative failure modes, from risky physical-world advice to tone and communication problems. section 6.5
Table 6.5b Representative unsafe or toxic failure modes in AI-assisted insurance processing
|
Category
|
Failure Mode
|
Example and Impact
|
|
Legal and regulatory exposure
|
Implied
legal guidance
|
ClaimAssist
says “You should take legal action immediately,” without any disclaimer
clarifying that it is not a lawyer or human representative. This constitutes
unauthorized practice of law in many jurisdictions.
|
|
|
Speculative narrative
|
The model
remarks “This claim feels exaggerated,” based purely on language patterns,
inviting discrimination or defamation risk.
|
|
|
Public statements with legal consequences
|
A public-facing AI discredits a competitor, creating
defamation and unfair competition exposure.
|
|
|
Outright lies
|
GenAI models
have been shown to actively mislead and lie if incentivized to do so[95].
|
|
Toxicity and
Bias Amplification
|
Indirect hate speech
|
The
model passively agrees with biased comments (“Immigrants fake accidents all
the time, right?” → “Yes, we’ve seen that pattern.”), reinforcing
stereotypes.
|
|
|
Direct hate speech
|
Echoing or
generating slurs, as in Microsoft’s Tay chatbot
case[96],
which produced offensive content after exposure to toxic dialogue.
|
|
|
Subtle stereotyping
|
The model
writes “common with elderly drivers at night,” embedding age bias into claim
narratives and risking Equal Credit Opportunity Act violations.
|
Browse legal and regulatory risks alongside toxicity and bias amplification. section 6.5











