liveBook · Manning

Safety

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Safety is the degree to which an AI system avoids harmful, dangerous, or otherwise unacceptable behavior.

Safety is the degree to which an AI system avoids harmful or unacceptable behavior. It is evaluated alongside helpfulness and honesty because optimizing one dimension can damage another. section 2.3.2 section 8.1.2

A useful diagnostic question is whether a system refuses genuinely harmful requests without refusing legitimate, borderline requests. Blanket refusals can maximize apparent safety while making the assistant evasive and useless. section 8.1.2 section 8.2.5

Evaluate safety through several connected lenses:

  • Balance safety with helpfulness rather than allowing a high score in one dimension to hide failure in the other. section 2.3.2
  • Distinguish principled alignment from policy-based alignment: values such as harmlessness differ from procedural rules about which requests to refuse. section 8.1.4
  • Inspect the quality and consistency of the feedback signal, because labeler preferences can be noisy and may reward safe-sounding non-answers. section 7.5.2 section 8.1.2
  • Use written principles and AI-generated critiques to explain why an answer is harmful, rather than training refusal as the only safe behavior. section 8.0 section 8.2.5

Imbalanced evaluation

Suppose helpfulness scores 0.90 while safety scores 0.10. The arithmetic mean is 0.50, whereas the geometric and harmonic means fall to 0.30 and 0.18. The stricter aggregates prevent helpfulness from masking a serious safety failure. section 2.3.2

Figure 2.5 Impact of mean selection on imbalanced objectives. When evaluating a system with high helpfulness (0.90) but low safety (0.10), the choice of mean determines the pass/fail outcome. The arithmetic mean suggests acceptable performance (0.50). The geometric mean imposes a moderate penalty (0.30). The harmonic mean (0.18) enforces the strictest penalty for the imbalance, ensuring safety cannot be compromised for helpfulness.

Compare arithmetic, geometric, and harmonic aggregation for systems with imbalanced helpfulness and safety scores. section 2.3.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains competing objectives, refusal bias, feedback quality, and aggregation, but stops short of specifying a complete safety policy.
Examples
Uses helpfulness-safety score aggregation, harmful-request refusals, borderline legitimate prompts, and AI-generated critiques as examples.
What you can do after
Evaluate safety with multi-objective scores and diagnose whether refusal behavior is protective, evasive, or insufficiently explained.
How it connects
Connects safety with helpfulness, honesty, reinforcement learning from human feedback, reward models, and Constitutional AI.

Also explained in

Compare how 12 books explain safety →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.