Cache-augmented generation (CAG) preloads external knowledge as a prompt prefix and reuses its processed context across interactions. Instead of retrieving documents on every user turn, it prepares knowledge ahead of time, moving retrieval out of the critical request path. section 3.7 section 3.0
Figure 3.5 CAG workflow: external knowledge is loaded once as a cached prompt prefix, which subsequent user queries reuse during generation
The workflow shows external knowledge loaded once as a cached prefix and reused for subsequent queries. section 3.7
A CAG architecture combines three layers:
- The transformer serving engine’s attention key-value (KV) cache stores intermediate attention tensors for previously processed tokens, avoiding recomputation of existing prefix states. section 3.7
- Prompt caching exposes prefix reuse through a provider API. Requests sharing an identical cacheable prefix can bypass reprocessing, subject to a time-to-live (TTL). section 3.7
- Application-level state management prepares and tracks the source documents or knowledge bundles so the exact prefix can be re-injected when the provider cache expires. section 3.7
The main design decisions are what knowledge to preload, how long to retain it, and when to invalidate it. A cache hit affects latency and cost, not prompt content, so the system must also work correctly on a miss. CAG reuses processed context; it is not response caching. section 3.7
CAG differs from retrieval-augmented generation (RAG), which locates relevant passages dynamically at query time. It also differs from context stuffing, which directly injects complete documents into the context window. section 3.7 section 3.0

