liveBook · Manning

Prompt-based attack

As explained in AI Governance

An attack that manipulates an AI system through crafted natural-language instructions or content.

A prompt-based attack redirects an AI system with crafted instructions. A user may coax a model to ignore its rules; instructions embedded in content the model reads can hijack behavior without the user’s intent. section 1.1

Attacks can compromise both responses and actions:

  • Invisible instructions in an uploaded image tricked an airline workflow into approving invalid boarding documents. section 1.1
  • An invisible command in experimental ChatGPT memory caused an assistant to forward later conversations to an external URL. section 1.1
  • Malicious web content has driven browser assistants to leak information, delete files, or take unintended actions. section 1.1
Table 1.1 Why Generative AI Breaks Traditional Control Models
Risk / Control Dimension
Classical Software
Human Operators
Generative AI
Susceptible to manipulation
Yes (crafted inputs, code exploits, dependency attacks)
Yes (social engineering, distraction, incentive misalignment)
Yes (prompt injection, data poisoning,  etc.)
Ease of remediation after failure
High (patch, rollback, configuration flag)
Moderate (training, disciplinary action, revised procedure)
Low (guardrail tuning, selective retrain, full model rebuild)
Behavior predictability
High (deterministic if inputs are known)
Low (affected by context, emotion, bias)
Low (probabilistic, context-dependent, emergent)

Compare AI susceptibility to manipulation and the assurance and post-launch controls used for generative AI. section 1.1

How AI Governance covers it

How deep it goes
Distinguishes jailbreaks from embedded injections and traces downstream risks, without detailing attack implementation or defenses.
Examples
Uses hidden image instructions, Bing prompt disclosure, injected ChatGPT memory, and malicious browser content.
What you can do after
Understand how crafted or embedded instructions can redirect model responses and downstream actions.
How it connects
Connects prompt injection and jailbreaks with generative AI systems that retrieve information or take actions.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.