Red-teaming

As explained in Building Reliable AI Systems

The practice of deliberately probing a system for weaknesses, unsafe behavior, or failure modes.

Red-teaming deliberately challenges an AI system to expose weaknesses that automated tests may miss. The aim is to find vulnerabilities before users do, not to break the system for its own sake. section 9.1.5

ShopBot failure probes

A red team might try to elicit an incorrect refund policy, an invented product, an impossible delivery promise, or internal business information. section 9.1.5

Use red-teaming as a repeating improvement cycle:

• Test adversarial prompts, extreme scenarios, and edge cases. section 9.1.5
• Analyze discovered weaknesses for their causes and implications. section 9.1.5
• Improve the system and test it again; repeat the cycle to build reliability. section 9.1.5

Red-teaming complements, rather than replaces, metrics and standard evaluations: each can reveal different failure modes. In production, combine evaluation approaches to assess reliability as systems and usage change. section 9.1.5 section 1.4.3

Figure 9.2 The red-teaming process for LLMs

See the testing, analysis, and refinement cycle. section 9.1.5

How Building Reliable AI Systems covers it

How deep it goes

Outlines adversarial, extreme, and edge-case testing followed by analysis and repeated improvement.

Examples

Tests a ShopBot for false refund policies, invented products, impossible delivery promises, and internal-information disclosure.

What you can do after

Diagnose vulnerabilities before users encounter them and guide repeated system improvements.

How it connects

Combines red-teaming with automated metrics and standard evaluations to reveal different failure modes.

liveBookmanning.com

Read Building Reliable AI Systems on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionA red-teaming protocol is a defined procedure for adversarially testing a system to discover vulnerabilities, unsafe behavior, or policy failures.
• 

  liveBookmanning.com
  Generative AI in ActionA red team is an authorized group that deliberately probes a system for vulnerabilities, failure modes, or harmful behavior.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA red team is an authorized person or group that tests a system by attempting adversarial actions against it.
• 

  liveBookmanning.com
  AI Agents in Action, Second EditionRed team testing is adversarial testing that probes a system with inputs or patterns intended to expose failures, vulnerabilities, or unsafe behavior.
• 

  liveBookmanning.com
  Machine Learning Platform EngineeringAdversarial testing is the deliberate evaluation of a system with challenging, manipulative, or harmful inputs to expose weaknesses.

Compare how 6 books explain red-teaming →

Related concepts

• adversarial inputuses · An input crafted, selected, or modified to exploit a system’s weaknesses or induce incorrect, unsafe, or otherwise unintended behavior.
• threat huntingcontrasts with · Threat hunting is the proactive investigation of systems and networks to discover malicious activity that automated detection may have missed.
• security controluses · A security control is a safeguard, policy, or restriction designed to reduce security risks or enforce security objectives.
• hypothesisused by · A hypothesis is a provisional proposition, explanation, or prediction that can be evaluated against evidence.

This explainer was generated by Manning's knowledge graph from the text of Building Reliable AI Systems. The section links point to the parts of the book it draws on.