liveBook · Manning

Risk

As explained in AI Governance

Risk is the possibility of an adverse outcome arising from uncertainty about an activity, system, or decision.

An AI assistant can turn a routine email into a security incident if it treats hidden instructions as legitimate commands. Risk is the possibility of an adverse outcome under uncertainty; in GenAI systems, it grows with the information a system can read and the actions it can take. section 4.0 section 8.4

Prompt filters and other guardrails can stop obvious attacks, but they are not enforced boundaries. Unlike conventional input validation, they cannot reliably separate instructions from data when both appear as plain text to a language model. section 4.0

Risk grows with capability

ClaimAssist began as a policy FAQ chatbot. Connecting customer data, adding record updates and request processing, and evaluating an agent that chooses tools each created new value—and new attack paths and failure modes. section 4.1

Figure 4.1 ClaimAssist v1 on the left is a simple policy chatbot. ClaimAssist v4 on the right can accept user uploaded images and change a claim status as “ready for adjuster review”.

Compare the initial policy chatbot with a version that accepts images and can change a claim’s status. section 4.1

Review each deployment before deciding where and under what conditions it can be used:

  • Identify what the system can read, including sensitive data and content an outsider can influence. section 8.4
  • Identify what it can do, which tools it can call, and whose authority it uses. section 8.4
  • Estimate the damage a failure could cause before detection, and check whether records would support a response. section 8.4
  • Set operating conditions: permitted data and tools, required approvals, records, and triggers for pausing or reviewing the deployment. section 8.4

Start with the least access the use case needs and expand only when production evidence supports it. Keep sensitive data out of reach where possible, route deterministic steps to non-AI systems, and record the limits of any delegated account access. section 8.4

How AI Governance covers it

How deep it goes
Provides review questions and operating conditions, while emphasizing that the goal is informed use rather than risk elimination.
Examples
Illustrates risk with a claims assistant’s expanding capabilities and the EchoLeak attack against Microsoft 365 Copilot.
What you can do after
Evaluate AI deployments by their access, actions, potential damage, and ability to support an incident response.
How it connects
Connects risk to prompt injection, access control, monitoring, and incident response.

Also explained in

Compare how 13 books explain risk →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.