liveBook · Manning

Red-teaming

As explained in AI Governance

Adversarial evaluation in which authorized testers simulate attackers to discover weaknesses, failures, or harmful behaviors.

Red-teaming tests a GenAI system by simulating adversarial behavior that routine QA may miss. Testers use varied inputs to expose weaknesses, failures, and subtle harms before release and after significant changes. section 6.5.2

A practical red-team cycle:

  • Assemble AI developers, security specialists, domain experts, and people able to identify subtle harms. section 6.5.2
  • Test structured harm categories with varied inputs, including emotionally charged language, filter-evasion requests, and jailbreak-like prompts. section 6.5.2
  • Document findings, refine prompts and classifiers, and add confirmed issues to a permanent regression suite. section 6.5.2
  • Schedule tests before major model updates and feature releases; assign an owner to triage issues for remediation or accepted risk. section 6.5.2

ClaimAssist

Red-team testing uncovered different follow-up questions based on names, fabricated repair estimates embedded in PDFs, and spelling variations that triggered stricter default handling. Standard QA had missed these issues. section 6.5.2

Before launch, confirm adversarial tests used the release candidate and checked for instruction overrides, data leaks, and response-policy bypasses. A completed conventional penetration test alone does not establish that these behaviors were tested. section 3.5

Figure 6.11 Red teaming workflow for GenAI systems

Use the workflow figure to orient red-team activities for GenAI systems. section 6.5.2

How AI Governance covers it

How deep it goes
Details team composition, adversarial inputs, finding triage, remediation, regression testing, and release timing.
Examples
Uses ClaimAssist cases involving name-based differences, fabricated PDF estimates, and provider-name spelling variations.
What you can do after
Practice adversarial evaluation, document findings, refine controls, and route issues for remediation or accepted risk.
How it connects
Connects adversarial testing with quality assurance, prompt and classifier refinement, and regression testing.

Also explained in

Compare how 8 books explain red-teaming →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.