An AI assistant can turn a routine email into a security incident if it treats hidden instructions as legitimate commands. Risk is the possibility of an adverse outcome under uncertainty; in GenAI systems, it grows with the information a system can read and the actions it can take. section 4.0 section 8.4
Prompt filters and other guardrails can stop obvious attacks, but they are not enforced boundaries. Unlike conventional input validation, they cannot reliably separate instructions from data when both appear as plain text to a language model. section 4.0
Risk grows with capability
ClaimAssist began as a policy FAQ chatbot. Connecting customer data, adding record updates and request processing, and evaluating an agent that chooses tools each created new value—and new attack paths and failure modes. section 4.1
Figure 4.1 ClaimAssist v1 on the left is a simple policy chatbot. ClaimAssist v4 on the right can accept user uploaded images and change a claim status as “ready for adjuster review”.

Compare the initial policy chatbot with a version that accepts images and can change a claim’s status. section 4.1
Review each deployment before deciding where and under what conditions it can be used:
- Identify what the system can read, including sensitive data and content an outsider can influence. section 8.4
- Identify what it can do, which tools it can call, and whose authority it uses. section 8.4
- Estimate the damage a failure could cause before detection, and check whether records would support a response. section 8.4
- Set operating conditions: permitted data and tools, required approvals, records, and triggers for pausing or reviewing the deployment. section 8.4
Start with the least access the use case needs and expand only when production evidence supports it. Keep sensitive data out of reach where possible, route deterministic steps to non-AI systems, and record the limits of any delegated account access. section 8.4












