A guardrail is a safety filter or control that detects, limits, or blocks inputs and outputs that violate defined restrictions. When Microsoft’s Bing chatbot produced unsettling responses, the company limited conversation length and restricted particular topics. section 1.19 section 3.5
Guardrails turn safety policies into operational controls. Policies commonly address racist or extremist content, legal and medical advice, and instructions for illegal or harmful actions, motivated by moral, reputational, legal, and regulatory concerns. section 3.5
Guardrails can operate at several points in an AI system:
- Post-process a generated response with a toxicity classifier before deciding whether to show it. section 3.6
- Constrain an agent’s available actions and require human approval before high-impact operations execute. section 8.20
- Monitor deployed agents with audit logs, anomaly detection, permission gating, and safeguards that adapt to observed behavior. section 8.24
Helpful versus harmless
A model may answer a request fluently while reinforcing sexist stereotypes or creating another form of harm. Blocking too aggressively can also make responses unhelpful, so guardrails must balance helpfulness, harmlessness, and honesty. section 3.5 section 3.6
Figure 3.1 The logic of a response decision with a post hoc toxicity classifier
Shows the decision logic for using a post hoc toxicity classifier to determine whether a generated response should be delivered. section 3.6
Guardrails reduce risk but do not guarantee safety. Indirect prompt injection can redirect tool-using agents through malicious webpages, files, or emails, while deciding what counts as risky remains difficult across changing domains. section 8.20 section 8.24
