liveBook · Manning

Cache-augmented generation

As explained in Context Engineering

Cache-augmented generation is a generation approach that preloads relevant knowledge and reuses its processed context across multiple interactions.

Cache-augmented generation (CAG) preloads external knowledge as a prompt prefix and reuses its processed context across interactions. Instead of retrieving documents on every user turn, it prepares knowledge ahead of time, moving retrieval out of the critical request path. section 3.7 section 3.0

Figure 3.5 CAG workflow: external knowledge is loaded once as a cached prompt prefix, which subsequent user queries reuse during generation
cag v8

The workflow shows external knowledge loaded once as a cached prefix and reused for subsequent queries. section 3.7

A CAG architecture combines three layers:

  • The transformer serving engine’s attention key-value (KV) cache stores intermediate attention tensors for previously processed tokens, avoiding recomputation of existing prefix states. section 3.7
  • Prompt caching exposes prefix reuse through a provider API. Requests sharing an identical cacheable prefix can bypass reprocessing, subject to a time-to-live (TTL). section 3.7
  • Application-level state management prepares and tracks the source documents or knowledge bundles so the exact prefix can be re-injected when the provider cache expires. section 3.7

The main design decisions are what knowledge to preload, how long to retain it, and when to invalidate it. A cache hit affects latency and cost, not prompt content, so the system must also work correctly on a miss. CAG reuses processed context; it is not response caching. section 3.7

CAG differs from retrieval-augmented generation (RAG), which locates relevant passages dynamically at query time. It also differs from context stuffing, which directly injects complete documents into the context window. section 3.7 section 3.0

How Context Engineering covers it

How deep it goes
Explains the KV-cache, prompt-caching, and application-state layers, including retention decisions and cache-miss behavior.
Examples
Illustrates external knowledge loaded as a cached prompt prefix and reused by subsequent user queries.
What you can do after
Evaluate CAG architecture by distinguishing cache layers and considering retention, invalidation, and cache misses.
How it connects
Connects prompt caching and KV caches with retrieval-augmented generation and context stuffing.

Also explained in

Compare how 2 books explain cache-augmented generation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Context Engineering. The section links point to the parts of the book it draws on.