Red-teaming tests a GenAI system by simulating adversarial behavior that routine QA may miss. Testers use varied inputs to expose weaknesses, failures, and subtle harms before release and after significant changes. section 6.5.2
A practical red-team cycle:
- Assemble AI developers, security specialists, domain experts, and people able to identify subtle harms. section 6.5.2
- Test structured harm categories with varied inputs, including emotionally charged language, filter-evasion requests, and jailbreak-like prompts. section 6.5.2
- Document findings, refine prompts and classifiers, and add confirmed issues to a permanent regression suite. section 6.5.2
- Schedule tests before major model updates and feature releases; assign an owner to triage issues for remediation or accepted risk. section 6.5.2
ClaimAssist
Red-team testing uncovered different follow-up questions based on names, fabricated repair estimates embedded in PDFs, and spelling variations that triggered stricter default handling. Standard QA had missed these issues. section 6.5.2
Before launch, confirm adversarial tests used the release candidate and checked for instruction overrides, data leaks, and response-policy bypasses. A completed conventional penetration test alone does not establish that these behaviors were tested. section 3.5
Figure 6.11 Red teaming workflow for GenAI systems

Use the workflow figure to orient red-team activities for GenAI systems. section 6.5.2







