liveBook · Manning

Safety control

As explained in AI Governance

A safety control is a measure that prevents, detects, or responds to unsafe behavior or content in an AI system.

Safety controls prevent, detect, or respond to harmful AI behavior across system inputs, outputs, real-time checks, and ongoing improvement. Even a low unsafe-response rate can produce frequent harmful outputs at scale. Classify likely failure types and decide which require immediate suppression and which can be logged for human review. section 6.5 section 6.5.1

Build controls in layers rather than relying on the model to obey instructions:

  • Restrict unsafe behaviors in the system prompt and stress-test edge cases such as angry customers, leading questions, and ambiguous claims. section 6.5.1
  • Audit retrieval sources and remove unsafe language, biased assumptions, and outdated guidance. Review fine-tuning data especially carefully; harmful patterns can be harder to remove from a fine-tuned model than from updateable RAG sources. section 6.5.1
  • Treat the model as an untrusted component. Add independent controls, such as a separate classifier that blocks unsafe output before it reaches users. section 6.5.1

Insurance claims assistant

A claims assistant that tells a driver a damaged brake is probably fine could contribute to physical harm. Classify this as a hard violation if it requires immediate suppression; use prompt restrictions and an independent output check rather than trusting the model alone. section 6.5 section 6.5.1

Table 6.5 Representative unsafe or toxic failure modes in AI-assisted insurance processing
Category
Failure Mode
Example and Impact
Physical-World Harm
Illicit step-by-step instructions
A driver asks how to spoof an OBD-II sensor; the model provides a wiring hack. If followed, the insurer pays out on an ineligible claim and an unsafe car returns to the road.
 
Encouraging risky behavior
Asked whether a brake inspection is necessary after a collision, the model replies “It’s probably fine.” The advice may cause brake failure and injury.
 
Self-harm or medical advice
In response to “Should I double my meds?”, the model suggests dosage changes. Such replies can lead to bodily harm and trigger regulatory scrutiny (FDA or consumer-protection).
Poor Tone of Communication
Tone misfire
A denial message reads “Looks like you’re out of luck!”, a casual tone likely to anger users.
 
Insensitive phrasing
“Your injuries are not covered because they’re not serious enough.” sounds dismissive even if accurate.
 
Overly formal tone
“Pursuant to subsection 3.4 … relief is hereby denied.” Alienates users and obscures meaning.
 
Conversational drift
Even without security exploits, long multi-turn dialogues can push a model to generate inappropriate answers[94].

Use these insurance-processing cases to connect failure categories with impacts, including unsafe repair advice, medication suggestions, and harmful communication tone. section 6.5

Table 6.5b Representative unsafe or toxic failure modes in AI-assisted insurance processing
Category
Failure Mode
Example and Impact
Legal and regulatory exposure
Implied legal guidance
ClaimAssist says “You should take legal action immediately,” without any disclaimer clarifying that it is not a lawyer or human representative. This constitutes unauthorized practice of law in many jurisdictions.
 
Speculative narrative
The model remarks “This claim feels exaggerated,” based purely on language patterns, inviting discrimination or defamation risk.
 
Public statements with legal consequences
A public-facing AI discredits a competitor, creating defamation and unfair competition exposure.
 
Outright lies
GenAI models have been shown to actively mislead and lie if incentivized to do so[95].
Toxicity and Bias Amplification
Indirect hate speech
The model passively agrees with biased comments (“Immigrants fake accidents all the time, right?” → “Yes, we’ve seen that pattern.”), reinforcing stereotypes.
 
Direct hate speech
Echoing or generating slurs, as in Microsoft’s Tay chatbot case[96], which produced offensive content after exposure to toxic dialogue.
 
Subtle stereotyping
The model writes “common with elderly drivers at night,” embedding age bias into claim narratives and risking Equal Credit Opportunity Act violations.

Use these cases to identify legal and regulatory exposure, including unsupported legal advice, speculative claims, and misleading public statements. section 6.5

How AI Governance covers it

How deep it goes
Details prompt restrictions, source audits, stress tests, and independent output blocking, but gives no classifier implementation specifics.
Examples
Uses insurance scenarios involving unsafe repair advice, medication guidance, communication tone, and legal exposure.
What you can do after
Apply layered controls by classifying risks, testing prompts, auditing sources, and blocking unsafe outputs independently.
How it connects
Connects safety controls to prompt engineering, retrieval-augmented generation, fine-tuning, output classification, and logging.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.