Red-teaming
Red-teaming is an adversarial assessment practice that deliberately simulates attacks or uses challenging inputs to identify a system’s vulnerabilities, failures, or unsafe behavior.
Introduction to Generative AI, Second Edition
A red-teaming protocol is a defined procedure for adversarially testing a system to discover vulnerabilities, unsafe behavior, or policy failures.
- How it is used
- Complements benchmarks by stress-testing AI systems for vulnerabilities and safety-mitigation failures.
- How deep it goes
- Outlines input creation, response checking, and vulnerability analysis, with an overview of manual and automated methods.
- Examples
- Illustrates testing with a sexist song prompt, hypothetical scenarios, unusual formatting, and possible agentic tool failures.
- What you can do after
- Evaluate and understand how adversarial inputs can expose model vulnerabilities and inform mitigation.
Read how Introduction to Generative AI, Second Edition explains red-teaming →
Generative AI in Action
A red team is an authorized group that deliberately probes a system for vulnerabilities, failure modes, or harmful behavior.
- How it is used
- Identifies application-specific harms and safety gaps as part of responsible AI risk assessment.
- How deep it goes
- Outlines planning, testing, recordkeeping, and follow-up, while distinguishing discovery from systematic measurement and mitigation.
- Examples
- Uses a customer service chatbot’s bias, privacy, and accuracy risks and names tools such as TextAttack and AI Fairness 360.
- What you can do after
- Evaluate and understand model and application risks, then use findings to guide measurement and mitigation.
Natural Language Processing in Action, Second Edition
A red team is an authorized person or group that tests a system by attempting adversarial actions against it.
- How it is used
- Uses authorized adversarial interaction to find LLM guardrail failures before harmful behavior reaches users.
- How deep it goes
- Explains roles, attack patterns, tester independence, and iterative defenses, but stops short of a complete testing protocol.
- Examples
- Uses adversarial prompt suffixes, unusual-request detectors, self-checks, and collected edge cases as testing artifacts.
- What you can do after
- Diagnose guardrail weaknesses and improve an LLM pipeline by converting adversarial failures into filters, checks, and regression cases.
Read how Natural Language Processing in Action, Second Edition explains red-teaming →
AI Agents in Action, Second Edition
Red team testing is adversarial testing that probes a system with inputs or patterns intended to expose failures, vulnerabilities, or unsafe behavior.
- How it is used
- Tests user-facing agents for safety breaks by probing terms, phrases, and patterns that may expose unsafe behavior.
- How deep it goes
- Defines adversarial safety testing and places it in an evaluation loop, but does not detail a full red-team procedure.
- Examples
- Uses targeted terms, phrases, and patterns as probes, then reruns tests to assess safety and behavioral stability.
- What you can do after
- Diagnose safety breaks in user-facing agents and feed the findings into iterative evaluation and improvement.
Read how AI Agents in Action, Second Edition explains red-teaming →
Machine Learning Platform Engineering
Adversarial testing is the deliberate evaluation of a system with challenging, manipulative, or harmful inputs to expose weaknesses.
- How it is used
- Strengthens evaluation and production hardening by exposing model errors and vulnerabilities in public-facing RAG and LLM systems.
- How deep it goes
- Connects robustness, edge cases, prompt injection, and vulnerability assessment, but supplies no detailed testing procedure.
- Examples
- Uses misclassified samples, edge-case simulations, harmful-content risks, and prompt-injection attacks as testing concerns.
- What you can do after
- Evaluate model robustness and LLM vulnerabilities within automated, tracked production-testing workflows.
Read how Machine Learning Platform Engineering explains red-teaming →
LLM Evaluation and Alignment, The Foundational Ideas
Red-team testing is adversarial testing in which evaluators deliberately seek ways to make a system fail or violate its requirements.
- How it is used
- Finds specification gaps and failure modes before deployment users encounter them.
- How deep it goes
- Explains a probe-record-update loop but stops short of prescribing a detailed red-team test suite.
- Examples
- Uses adversarial prompts, deployed-model probes, uncovered failure modes, and new specification clauses as concrete artifacts.
- What you can do after
- Diagnose uncovered model failures and convert them into specification changes and subsequent alignment work.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains red-teaming →
AI Governance
Adversarial evaluation in which authorized testers simulate attackers to discover weaknesses, failures, or harmful behaviors.
- How it is used
- Evaluates GenAI systems for adversarial failures that routine QA may miss and supplies evidence for launch readiness.
- How deep it goes
- Details team composition, adversarial inputs, finding triage, remediation, regression testing, and release timing.
- Examples
- Uses ClaimAssist cases involving name-based differences, fabricated PDF estimates, and provider-name spelling variations.
- What you can do after
- Practice adversarial evaluation, document findings, refine controls, and route issues for remediation or accepted risk.
Building Reliable AI Systems
The practice of deliberately probing a system for weaknesses, unsafe behavior, or failure modes.
- How it is used
- Finds AI-system vulnerabilities and unexpected failures before they reach users.
- How deep it goes
- Outlines adversarial, extreme, and edge-case testing followed by analysis and repeated improvement.
- Examples
- Tests a ShopBot for false refund policies, invented products, impossible delivery promises, and internal-information disclosure.
- What you can do after
- Diagnose vulnerabilities before users encounter them and guide repeated system improvements.
Read how Building Reliable AI Systems explains red-teaming →
Also covered in
- AI Agents for Offensive SecurityIn early access (MEAP)
- LLM Customization and Fine-TuningIn early access (MEAP)
- Machines that ThinkIn early access (MEAP)
- Cyber Threat Hunting
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- biasBias is a systematic tendency or deviation that distorts measurements, estimates, judgments, or outcomes relative to a relevant reference or target.
- generative modelingGenerative modeling is a machine-learning approach that learns patterns or a data distribution to generate new samples resembling the data.
- guardrailA guardrail is a constraint, control, or monitoring mechanism that limits system behavior to reduce risks and keep actions or outputs within acceptable bounds.
Related concepts
- adversarial inputuses · An input crafted, selected, or modified to exploit a system’s weaknesses or induce incorrect, unsafe, or otherwise unintended behavior.
- hypothesisused by · A hypothesis is a proposed proposition, explanation, prediction, or model whose validity or plausibility can be assessed against evidence.
- evaluationcontrasts with · Evaluation is the systematic assessment of a system, model, process, or outcome against relevant criteria or objectives using evidence.
- offensive securitypart of · Offensive security is the practice of proactively simulating or conducting attacks to discover and exploit weaknesses in systems and networks.
- Promptfooused by · Promptfoo is an open-source tool for testing, evaluating, and comparing prompts and large language model applications.
Questions about red-teaming
What is red-teaming?
Red-teaming is an adversarial assessment practice that deliberately simulates attacks or uses challenging inputs to identify a system’s vulnerabilities, failures, or unsafe behavior.
Which Manning books explain red-teaming?
Introduction to Generative AI, Second Edition, Generative AI in Action, Natural Language Processing in Action, Second Edition, AI Agents in Action, Second Edition, Machine Learning Platform Engineering, LLM Evaluation and Alignment, The Foundational Ideas, AI Governance and Building Reliable AI Systems.
What should I know before learning red-teaming?
Manning books that teach red-teaming usually introduce language model, bias, generative modeling and guardrail first.
What is the difference between red-teaming and blue team?
Red-teaming is an adversarial assessment practice that deliberately simulates attacks or uses challenging inputs to identify a system’s vulnerabilities, failures, or unsafe behavior. A blue team is a cybersecurity group responsible for defending systems by monitoring, detecting, preventing, responding to, and mitigating attacks.
What is red-teaming part of?
Red-teaming is part of offensive security. Offensive security is the practice of proactively simulating or conducting attacks to discover and exploit weaknesses in systems and networks.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.







