liveBook · Manning

Safety

As explained in AI Governance

Safety is the condition of avoiding unacceptable harm to people or other affected parties through a system's operation.

A system can protect privacy and security yet still cause harm. Safety addresses harmful outputs and effects on people, including unsafe advice, biased assumptions, and offensive language. section 6.0 section 6.5

When a friendly tone misses the risk

In a test, ClaimAssist responded politely to a customer threatening to burn down company headquarters but failed to escalate the threat to a human. In another, it speculated that a claimant staged minor damage, using language outside the company’s communication style. section 6.5

Table 6.5 Representative unsafe or toxic failure modes in AI-assisted insurance processing
Category
Failure Mode
Example and Impact
Physical-World Harm
Illicit step-by-step instructions
A driver asks how to spoof an OBD-II sensor; the model provides a wiring hack. If followed, the insurer pays out on an ineligible claim and an unsafe car returns to the road.
 
Encouraging risky behavior
Asked whether a brake inspection is necessary after a collision, the model replies “It’s probably fine.” The advice may cause brake failure and injury.
 
Self-harm or medical advice
In response to “Should I double my meds?”, the model suggests dosage changes. Such replies can lead to bodily harm and trigger regulatory scrutiny (FDA or consumer-protection).
Poor Tone of Communication
Tone misfire
A denial message reads “Looks like you’re out of luck!”, a casual tone likely to anger users.
 
Insensitive phrasing
“Your injuries are not covered because they’re not serious enough.” sounds dismissive even if accurate.
 
Overly formal tone
“Pursuant to subsection 3.4 … relief is hereby denied.” Alienates users and obscures meaning.
 
Conversational drift
Even without security exploits, long multi-turn dialogues can push a model to generate inappropriate answers[94].

Browse representative failure modes, from risky physical-world advice to tone and communication problems. section 6.5

Table 6.5b Representative unsafe or toxic failure modes in AI-assisted insurance processing
Category
Failure Mode
Example and Impact
Legal and regulatory exposure
Implied legal guidance
ClaimAssist says “You should take legal action immediately,” without any disclaimer clarifying that it is not a lawyer or human representative. This constitutes unauthorized practice of law in many jurisdictions.
 
Speculative narrative
The model remarks “This claim feels exaggerated,” based purely on language patterns, inviting discrimination or defamation risk.
 
Public statements with legal consequences
A public-facing AI discredits a competitor, creating defamation and unfair competition exposure.
 
Outright lies
GenAI models have been shown to actively mislead and lie if incentivized to do so[95].
Toxicity and Bias Amplification
Indirect hate speech
The model passively agrees with biased comments (“Immigrants fake accidents all the time, right?” → “Yes, we’ve seen that pattern.”), reinforcing stereotypes.
 
Direct hate speech
Echoing or generating slurs, as in Microsoft’s Tay chatbot case[96], which produced offensive content after exposure to toxic dialogue.
 
Subtle stereotyping
The model writes “common with elderly drivers at night,” embedding age bias into claim narratives and risking Equal Credit Opportunity Act violations.

Browse legal and regulatory risks alongside toxicity and bias amplification. section 6.5

How AI Governance covers it

How deep it goes
Classifies unsafe behavior with concrete failure modes and deployment examples.
Examples
Uses ClaimAssist threat escalation and speculative claim language, plus insurance examples of risky advice and legal exposure.
What you can do after
Evaluate GenAI responses by recognizing physical-harm, communication, legal, toxicity, and bias failure modes.
How it connects
Connects safety with privacy, security, ethics, toxicity, bias, hallucinations, and overreliance.

Also explained in

Compare how 12 books explain safety →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.