liveBook · Manning

Attack surface

As explained in AI Governance

The set of points through which an attacker can attempt to affect or compromise a system.

An attack surface is the set of points through which an attacker can attempt to affect or compromise a system. In a generative AI workflow, it includes not only surrounding software but also the model and the content it reads. Data access, write access, and the ability to act expand what an attacker may be able to influence. section 4.0 section 1.1

A language model interprets instructions and reference material in natural language rather than enforcing a reliable technical boundary between them. An attacker can exploit that ambiguity by placing instructions in content the assistant is supposed to process. Filters can stop obvious attempts, but they are not an enforced boundary. section 4.0 section 4.6.3

A poisoned retrieval path

An attacker who can place a document or email where an assistant searches may hide instructions in it. If the assistant retrieves that content, the instructions can influence its response or actions. The combination of private-data access, untrusted content, and an outward communication path can enable exfiltration. section 4.6.3

Match controls to the exposed capabilities. Narrow search scope and exclude or de-prioritize loosely governed sources; sanitize documents before indexing to remove hidden formatting and metadata; and require source citations so unexpected inputs are visible. Policy engines, tiered approval, and sandboxing can constrain actions as capabilities expand. These measures reduce exposure and influence rather than guarantee that retrieved instructions will be ignored. section 4.6.3 section 4.0

How AI Governance covers it

How deep it goes
Explains instruction-data ambiguity and retrieval poisoning, then identifies scope, sanitization, citation, and action controls without guaranteeing prevention.
Examples
Uses hidden instructions in searchable documents and emails, and an AI document-checking workflow that approves invalid documents.
What you can do after
Evaluate/understand how added access and agency change exposure, and select controls to reduce retrieval and action risks.
How it connects
Connects prompt injection and document poisoning to retrieval, exfiltration, guardrails, and sandboxing.

Also explained in

Compare how 7 books explain attack surface →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.