liveBook · Manning

Threat model

As explained in AI Governance

A structured technique for identifying, analyzing, and prioritizing threats to a system.

A threat model is a written list of how an AI system could be attacked or fail. Teams use it during design review to identify risks before release. section 3.1 section 1.5

Threat modeling can surface attack paths such as:

  • Data poisoning: someone contaminates training data. section 1.5
  • Model inversion: outputs might reveal sensitive information. section 1.5
  • Jailbreaking: a user tries to make the model produce harmful content. section 1.5

Security architects and privacy engineers use the findings to refine system design, including privacy-by-design controls, detailed logging, data minimization, and encryption for stored data. section 1.5

Table 3.1 Six Levels of GenAI Governance (6L-G), with decision owners, key artifacts, and ISO 42001 mapping
Level
Who is involved
Key artifacts
ISO 42001 phase
1. Strategy & Policy
Executives, a GenAI steering committees, policy managers, GRC specialists
Responsible AI charter, acceptable use policy, risk tiers, AI inventory
Plan
2. Risk & Impact Assessment
Legal, compliance, product, risk
AI impact assessments, vendor review reports, contract terms for data use, retention, logging, and incident notice
Plan
3. Design and Implementation Review
Security, privacy, engineering, data science
Data flow diagram, threat model, onboarding status to technical tools (e.g. guardrails, AI gateways), code review and code analysis result,  credential storage and access limits, vetted retrieval sources, structured logging
Plan and Do
4. Launch Readiness Review
QA, security, privacy, product, domain experts, data science, RAI Office
Release test plan, tested version record, security/privacy/adversarial test results, model performance evaluation, launch approval
Do
5. Operations & Monitoring
MLOps, security operations, privacy officers, incident response
Logs of prompts, responses, and tool calls, alerts for prompt injection and jailbreak attempts, output quality and drift monitoring, incident response plan, record of which model and prompt version is live
Do and Check
6. Learning & Improvement
Governance boards, auditors, risk owners
Post incident report, internal audit report, updated policy
Act

See where the threat model fits among the artifacts and responsibilities for design and implementation review. section 3.1

How AI Governance covers it

How deep it goes
Defines a threat model as a written list of deficiencies and names attack paths such as poisoning, inversion, and jailbreaking.
Examples
Names data poisoning, model inversion, and jailbreaking as threats to examine.
What you can do after
Evaluate system designs by identifying plausible attacks and failures that inform controls.
How it connects
Connects threat modeling to privacy-by-design, data minimization, logging, encryption, and red teaming.

Also explained in

Compare how 11 books explain threat model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.