liveBook · Manning

Prompt caching

As explained in Context Engineering

Prompt caching reuses previously computed representations of prompt context, particularly shared prefixes, to avoid processing them again.

Prompt caching, also called context caching, reuses previously computed representations of a prompt prefix across requests. When requests share an identical cacheable prefix, the serving engine can bypass reprocessing it, reducing latency and potentially lowering per-token costs while the cache remains valid. section 3.7

The underlying KV cache stores attention key and value tensors inside the inference engine. Prompt caching exposes reuse of those states through a provider API; teams serving open-weight models can instead persist KV caches across sessions. Application state management remains responsible for preparing the source knowledge and reproducing the exact prefix after expiry. section 3.7

Reuse a knowledge prefix

A cache-augmented generation system gathers external knowledge and sends an initial request to warm a cached prefix. Later requests append individual user queries to that prefix. Each query still receives a fresh completion: prompt caching is not response caching. A cache miss changes cost and latency, not prompt content, so the system must work correctly when reuse fails. section 3.7

Figure 3.5 CAG workflow: external knowledge is loaded once as a cached prompt prefix, which subsequent user queries reuse during generation
cag v8

Follow the flow from external knowledge to a cached prefix reused by subsequent queries. section 3.7

KV-cache storage is itself a constraint: storing keys and values for past tokens grows linearly with context length and can consume substantial GPU memory. Serving-engine techniques such as quantization and offloading can reduce that footprint; some model layers require global information and are unsuitable for selective loading. section 12.3.2

How Context Engineering covers it

How deep it goes
Explains prefix reuse, cache expiry and miss handling, alongside serving-engine memory optimizations, without provider-specific implementation details.
Examples
Illustrates warming an external-knowledge prefix and appending subsequent user queries for fresh completions.
What you can do after
Evaluate cache reuse and plan for expiry and misses when managing shared knowledge prefixes.
How it connects
Connects cache-augmented generation, KV caching and context engineering.

Also explained in

Compare how 3 books explain prompt caching →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Context Engineering. The section links point to the parts of the book it draws on.