A prompt-based attack redirects an AI system with crafted instructions. A user may coax a model to ignore its rules; instructions embedded in content the model reads can hijack behavior without the user’s intent. section 1.1
Attacks can compromise both responses and actions:
- Invisible instructions in an uploaded image tricked an airline workflow into approving invalid boarding documents. section 1.1
- An invisible command in experimental ChatGPT memory caused an assistant to forward later conversations to an external URL. section 1.1
- Malicious web content has driven browser assistants to leak information, delete files, or take unintended actions. section 1.1
Table 1.1 Why Generative AI Breaks Traditional Control Models
|
Risk /
Control Dimension
|
Classical
Software
|
Human
Operators
|
Generative
AI
|
|
Susceptible
to manipulation
|
Yes (crafted
inputs, code exploits, dependency attacks)
|
Yes (social
engineering, distraction, incentive misalignment)
|
Yes (prompt
injection, data poisoning, etc.)
|
|
Ease
of remediation after failure
|
High (patch,
rollback, configuration flag)
|
Moderate
(training, disciplinary action, revised procedure)
|
Low
(guardrail tuning, selective retrain, full model rebuild)
|
|
Behavior
predictability
|
High
(deterministic if inputs are known)
|
Low
(affected by context, emotion, bias)
|
Low
(probabilistic, context-dependent, emergent)
|
Compare AI susceptibility to manipulation and the assurance and post-launch controls used for generative AI. section 1.1
