Safety
Safety is the property of avoiding specified harmful, hazardous, invalid, or otherwise forbidden outcomes under relevant conditions.
Sutskever's List
AGI safety is the field concerned with preventing artificial general intelligence from causing unintended, harmful, or uncontrolled outcomes.
- How it is used
- Frames both present-day AI harms and precautionary decisions about more capable systems.
- How deep it goes
- Defines unsafe systems through possible sources of harm and works through incidents and decisions, but gives no technical safety procedure.
- Examples
- Uses an airline chatbot’s fare advice, an automated-driving failure, GPT-2’s release debate, and the Altman firing.
- What you can do after
- Evaluate/understand it by distinguishing current harms from speculative risks and recognizing the costs of preemptive safety decisions.
LLM Evaluation and Alignment, The Foundational Ideas
Safety is the degree to which an AI system avoids harmful, dangerous, or otherwise unacceptable behavior.
- How it is used
- Evaluates whether alignment preserves harmless behavior without sacrificing helpfulness or producing broad, evasive refusal.
- How deep it goes
- Explains competing objectives, refusal bias, feedback quality, and aggregation, but stops short of specifying a complete safety policy.
- Examples
- Uses helpfulness-safety score aggregation, harmful-request refusals, borderline legitimate prompts, and AI-generated critiques as examples.
- What you can do after
- Evaluate safety with multi-objective scores and diagnose whether refusal behavior is protective, evasive, or insufficiently explained.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains safety →
Rust Servers, Services, and Apps
The property of software or a programming language preventing classes of invalid or hazardous behavior.
- How it is used
- Frames program safety as a benefit of Rust for application backends and software infrastructure.
- How deep it goes
- Distinguishes three safety aspects and explains compile-time type checking, while deferring detailed memory-safety analysis.
- Examples
- Contrasts manual memory management in C and C++ with garbage-collected languages including Java, C#, Python, Ruby, and Go.
- What you can do after
- Understand how compile-time type checking helps prevent a class of runtime errors.
Think Distributed Systems
A system guarantee that a specified bad event or condition will never occur.
- How it is used
- Uses safety properties to specify forbidden system behaviors and judge correctness in distributed transactions, failure handling, and consensus.
- How deep it goes
- Defines safety formally, models reachable states, and applies it to transactions and consensus, while separating it from liveness and system-model limits.
- Examples
- Illustrates safety with a two-participant transaction, a state variable constrained to 1 or 2, coordinator failure, and consensus decision rules.
- What you can do after
- Evaluate distributed-system guarantees by identifying forbidden states, conflicting decisions, failure responses, and the boundary between safety and liveness.
Introduction to Generative AI, Second Edition
Harmlessness is the property of an AI system's behavior that avoids causing or enabling harm.
- How it is used
- Frames AI behavior around preventing unwanted outputs and managing harms in language-model and agent deployments.
- How deep it goes
- Explains safeguards, filtering, oversight, and trade-offs, but stops short of implementation detail or a complete theory of advanced-agent safety.
- Examples
- Uses toxic or sexist outputs, response blocking, toxicity classification, and supervised agents making safety-sensitive decisions.
- What you can do after
- Evaluate safety trade-offs between useful responses, harmful outputs, filtering errors, and human control.
Read how Introduction to Generative AI, Second Edition explains safety →
Vibe Engineering
Safety is the prevention of harm to people, systems, or the environment from a system's operation or failure.
- How it is used
- Constrains probabilistic systems through refusal rules, capability limits, and controlled tool-use blast radius.
- How deep it goes
- Defines safety through trade-offs and enforceable specifications, stopping short of detailed implementation patterns.
- Examples
- Uses injection resistance, tool-use blast-radius limits, executable specifications, and CI/CD verification gates as examples.
- What you can do after
- Apply safety through executable rules and repeated verification rather than prompts, temperature settings, or model upgrades alone.
AI Governance
Safety is the condition of avoiding unacceptable harm to people or other affected parties through a system's operation.
- How it is used
- Distinguishes safety from privacy and security by focusing on harmful outputs and their effects on people.
- How deep it goes
- Classifies unsafe behavior with concrete failure modes and deployment examples.
- Examples
- Uses ClaimAssist threat escalation and speculative claim language, plus insurance examples of risky advice and legal exposure.
- What you can do after
- Evaluate GenAI responses by recognizing physical-harm, communication, legal, toxicity, and bias failure modes.
The Coder Cafe
A safety property is a condition stating that an undesirable event never occurs during a system's execution.
- How it is used
- Frames correctness guarantees in concurrent and distributed systems, especially databases, consensus, and leader election.
- How deep it goes
- Defines safety, contrasts it with liveness, and applies it to mutexes, leader election, consensus, and network partitions.
- Examples
- Uses mutex deadlock prevention, single-leader election, and consensus behavior during network partitions.
- What you can do after
- Evaluate/understand whether a system prevents forbidden events and how safety choices affect progress during failures.
Generative AI in Action
AI safety concerns the prevention and mitigation of harmful, unreliable, or uncontrollable behavior by AI systems.
- How it is used
- Frames safety as a deployment concern requiring validation, privacy protection, fairness, and proactive prevention of misuse.
- How deep it goes
- Organizes safety into four categories and identifies concrete harms, but does not develop detailed implementation procedures or mathematical methods.
- Examples
- Uses RAG grounding, controlled enterprise data, source attribution, privacy compliance, deepfakes, cyberattacks, misinformation, and incorrect information.
- What you can do after
- Evaluate/understand AI risks and apply grounded data, validation, fairness analysis, and misuse controls to enterprise deployments.
Natural Language Processing in Action, Second Edition
AI safety is the field concerned with preventing artificial intelligence systems from causing harmful outcomes and maintaining reliable human control, including over long-term risks.
- How it is used
- Frames harmful autonomous behavior as a control problem requiring attention to both current failures and future existential risks.
- How deep it goes
- Defines the safety–ethics distinction and illustrates harms, but stops short of presenting technical mitigation methods.
- Examples
- Uses harmful dieting advice, deceptive airline interactions, attention-driven recommendations, geopolitical conflict, and human disempowerment as risk examples.
- What you can do after
- Evaluate AI harms by distinguishing immediate ethical failures from long-term control and existential-safety concerns.
Read how Natural Language Processing in Action, Second Edition explains safety →
Rust in Action: Systems programming concepts and techniques
Software safety is the property of software behavior being constrained to avoid invalid, harmful, or erroneous outcomes.
- How it is used
- Frames Rust’s central promise: preventing invalid behavior while retaining control and systems-level performance.
- How deep it goes
- Explains compile-time memory safety, concurrency errors, and zero-cost checks, but does not detail the underlying checking mechanisms.
- Examples
- Uses invalid data access, insecure IoT input parsing, infrequently updated firmware, and multiple integer representations.
- What you can do after
- Evaluate how Rust’s compile-time guarantees protect memory and concurrency without requiring runtime safety costs.
Read how Rust in Action: Systems programming concepts and techniques explains safety →
The Art of Code
A safety guarantee is an assurance that specified invalid or unsafe program conditions are prevented or detected.
- How it is used
- Evaluates code by how early it exposes missing cases and invalid assumptions, preferably through compiler support.
- How deep it goes
- Compares runtime-risk reduction and compiler-checked exhaustiveness, stopping short of treating safety as an isolated quality.
- Examples
- Contrasts Kotlin if/else promotional-message selection with Haskell algebraic data types and pattern matching.
- What you can do after
- Evaluate alternative implementations by their handling of invalid assumptions, missing variants, casts, and compiler-checkable cases.
Also covered in
- Context EngineeringIn early access (MEAP)
- Real-World Cryptography
- Build AI Drug Discovery PipelinesIn early access (MEAP)
- Learn Haskell by Example
- AI Agents in Action, Second Edition
- Fight Fraud with Machine LearningIn early access (MEAP)
- Evaluating AI SystemsIn early access (MEAP)
- Transformers in Action
- Digital Twins in Action
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- promptA prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
- AI agentAn AI agent is a computational system that uses artificial intelligence to interpret information and select or perform actions toward goals while interacting with an environment.
- artificial intelligenceArtificial intelligence is the field of computing concerned with creating systems that perform tasks associated with intelligence, such as learning, reasoning, perception, and decision-making.
Related concepts
- state machine safetyincludes · State machine safety is the property that no two processes apply different log entries at the same index to their state machines.
- helpfulnesscontrasts with · Helpfulness is the degree to which a system response provides relevant and useful assistance toward a user’s request or goal.
- livenesscontrasts with · Liveness is a correctness property requiring that a specified desirable event or system progress eventually occurs.
- AI control problemequivalent to · The AI control problem is the challenge of keeping artificial intelligence systems controllable by humans and preventing harmful unintended behavior.
Questions about safety
What is safety?
Safety is the property of avoiding specified harmful, hazardous, invalid, or otherwise forbidden outcomes under relevant conditions.
Which Manning books explain safety?
Sutskever's List, LLM Evaluation and Alignment, The Foundational Ideas, Rust Servers, Services, and Apps, Think Distributed Systems, Introduction to Generative AI, Second Edition, Vibe Engineering, AI Governance, The Coder Cafe and 4 more.
What should I know before learning safety?
Manning books that teach safety usually introduce language model, prompt, AI agent and artificial intelligence first.
What is the difference between safety and helpfulness?
Safety is the property of avoiding specified harmful, hazardous, invalid, or otherwise forbidden outcomes under relevant conditions. Helpfulness is the degree to which a system response provides relevant and useful assistance toward a user’s request or goal.
What kinds of safety are there?
Kinds of safety include state machine safety and agreement.
What is safety also called?
Safety is also called harmlessness.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.











