An assistant that drafts emails becomes a different security problem when it can also send them. If an attacker puts instructions in an incoming email, the assistant might treat that text as a command rather than as content to read. With access to sending tools, the result could be an unauthorized message to the user’s contacts. This is prompt injection: crafted input that tries to make an LLM disregard its intended instructions or take an unauthorized action. section 6.3.6 section 3.2.9
The stakes depend on what the model can access and do. A successful attack might bypass a content rule, expose an API key or environment variable, or trigger code execution in an unprotected environment. An agent with tools or plugins adds another route to harm because its responses can become actions. Sensitive company data also deserves attention: information submitted to a service, used in training, or made accessible to an assistant may create exposure risks. Check the provider’s current data-use terms rather than assuming every service retains and trains on submitted data. section 3.2.9 section 6.3.6 section 1.2.2
A convincing answer is not proof of a leak
A direct request for a secret cola recipe produced a refusal; a trickier prompt produced a plausible recipe. The resulting drink did not taste like the claimed product. The example illustrates both instruction bypass and a diagnostic trap: LLMs can invent convincing secrets, so an apparent disclosure is not, by itself, evidence that a real secret was recovered. section 3.2.9
For an LLM service, use overlapping precautions rather than treating one filter as a guarantee:
- Validate and sanitize incoming text, including content the assistant may encounter while using tools. section 6.3.6
- Check prompts for prohibited requests and outputs for unwanted disclosures; keyword search or a second LLM can support context-aware filtering. section 6.3.6
- Monitor for attacks, update models regularly, and train them to refuse inappropriate requests. Expect determined attackers to work around filtering. section 6.3.6
The practical question is not simply whether a model will refuse a malicious prompt. Ask what information it can see, what actions it can execute, and how you will detect a failure. Prompt injection matters most when an instruction bypass crosses one of those boundaries. section 3.2.9 section 6.3.6
