Red-teaming
Red-teaming is an adversarial assessment practice in which evaluators simulate attacks or use challenging inputs to identify a system’s weaknesses, failures, or unsafe behavior.
Introduction to Generative AI, Second Edition
A red-teaming protocol is a defined procedure for adversarially testing a system to discover vulnerabilities, unsafe behavior, or policy failures.
- How it is used
- Uses simulated attacks to uncover LLM vulnerabilities and test safeguards alongside standard benchmarks.
- How deep it goes
- Defines the practice and outlines input creation, response checking, and pattern-finding; describes automated prompt generation without implementation details.
- Examples
- Illustrates failures with a persona prompt producing sexist stereotypes and compares unusual formatting with hypothetical scenarios.
- What you can do after
- Practice/perform basic adversarial probing by submitting difficult inputs, checking responses, and identifying recurring weaknesses for mitigation.
Read how Introduction to Generative AI, Second Edition explains red-teaming →
Generative AI in Action
A red team is an authorized group that deliberately probes a system for vulnerabilities, failure modes, or harmful behavior.
- How it is used
- Uses deliberate probing to identify LLM vulnerabilities and potential harms within a responsible AI lifecycle.
- How deep it goes
- Details planning, testing, recording, and follow-through, but stops short of a complete attack procedure.
- Examples
- Uses customer-service chatbot risks and names adversarial-input, fairness-audit, and explainability tools.
- What you can do after
- Practice/perform planned model-and-application probes, record failures, and turn findings into measurement and mitigation work.
Natural Language Processing in Action, Second Edition
A red team is an authorized person or group that tests a system by attempting adversarial actions against it.
- How it is used
- Uses authorized adversarial interaction to find LLM guardrail failures before harmful behavior reaches users.
- How deep it goes
- Explains roles, attack patterns, tester independence, and iterative defenses, but stops short of a complete testing protocol.
- Examples
- Uses adversarial prompt suffixes, unusual-request detectors, self-checks, and collected edge cases as testing artifacts.
- What you can do after
- Diagnose guardrail weaknesses and improve an LLM pipeline by converting adversarial failures into filters, checks, and regression cases.
Read how Natural Language Processing in Action, Second Edition explains red-teaming →
AI Agents in Action, Second Edition
Red team testing is adversarial testing that probes a system with inputs or patterns intended to expose failures, vulnerabilities, or unsafe behavior.
- How it is used
- Tests user-facing agents for safety breaks by probing terms, phrases, and patterns that may expose unsafe behavior.
- How deep it goes
- Defines adversarial safety testing and places it in an evaluation loop, but does not detail a full red-team procedure.
- Examples
- Uses targeted terms, phrases, and patterns as probes, then reruns tests to assess safety and behavioral stability.
- What you can do after
- Diagnose safety breaks in user-facing agents and feed the findings into iterative evaluation and improvement.
Read how AI Agents in Action, Second Edition explains red-teaming →
Machine Learning Platform Engineering
Adversarial testing is the deliberate evaluation of a system with challenging, manipulative, or harmful inputs to expose weaknesses.
- How it is used
- Strengthens evaluation and production hardening by exposing model errors and vulnerabilities in public-facing RAG and LLM systems.
- How deep it goes
- Connects robustness, edge cases, prompt injection, and vulnerability assessment, but supplies no detailed testing procedure.
- Examples
- Uses misclassified samples, edge-case simulations, harmful-content risks, and prompt-injection attacks as testing concerns.
- What you can do after
- Evaluate model robustness and LLM vulnerabilities within automated, tracked production-testing workflows.
Read how Machine Learning Platform Engineering explains red-teaming →
Building Reliable AI Systems
The practice of deliberately probing a system for weaknesses, unsafe behavior, or failure modes.
- How it is used
- Finds AI-system vulnerabilities and unexpected failures before they reach users.
- How deep it goes
- Outlines adversarial, extreme, and edge-case testing followed by analysis and repeated improvement.
- Examples
- Tests a ShopBot for false refund policies, invented products, impossible delivery promises, and internal-information disclosure.
- What you can do after
- Diagnose vulnerabilities before users encounter them and guide repeated system improvements.
Read how Building Reliable AI Systems explains red-teaming →
Related concepts
- adversarial inputuses · An adversarial input is an input deliberately or effectively selected or modified to exploit a system’s weaknesses or induce incorrect, unsafe, or otherwise unintended behavior.
- security controluses · A security control is a safeguard, policy, procedure, or mechanism intended to prevent, detect, limit, or respond to security risks.
- Promptfooused by · Promptfoo is an open-source tool for testing, evaluating, and comparing prompts and large language model applications.
- evaluationcontrasts with · Evaluation is the systematic assessment of a system, result, or approach against relevant criteria, evidence, or objectives.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.