Red-teaming is adversarial testing: probe a system with terms, phrases, or patterns intended to expose failures, vulnerabilities, or unsafe behavior. Any user-facing agent or agent system should undergo some form of it. section 7.1
Treat red-teaming as one part of a broader evaluation system. Benchmark testing measures known goals, human feedback captures real-world reactions, and automated evaluators judge complex outputs; red-teaming specifically searches for safety breaks. section 7.1 section 7.6
Use red-teaming to strengthen an agent evaluation loop:
- Probe with targeted terms, phrases, or patterns that could break the system. section 7.1
- Treat discovered safety breaks as evaluation results rather than isolated incidents. section 7.6
- Combine red-team results with benchmarks, human feedback, and automated evaluation when appropriate. section 7.1 section 7.6
- Rerun tests for consistency because agent behavior can vary across attempts. section 7.6
A basic red-team probe
For a user-facing agent, submit a deliberately chosen phrase or pattern intended to trigger unsafe behavior. Record whether the system breaks, then include the result in repeated evaluation and improvement cycles. section 7.1 section 7.6
