Prompt caching, also called context caching, reuses previously computed representations of a prompt prefix across requests. When requests share an identical cacheable prefix, the serving engine can bypass reprocessing it, reducing latency and potentially lowering per-token costs while the cache remains valid. section 3.7
The underlying KV cache stores attention key and value tensors inside the inference engine. Prompt caching exposes reuse of those states through a provider API; teams serving open-weight models can instead persist KV caches across sessions. Application state management remains responsible for preparing the source knowledge and reproducing the exact prefix after expiry. section 3.7
Reuse a knowledge prefix
A cache-augmented generation system gathers external knowledge and sends an initial request to warm a cached prefix. Later requests append individual user queries to that prefix. Each query still receives a fresh completion: prompt caching is not response caching. A cache miss changes cost and latency, not prompt content, so the system must work correctly when reuse fails. section 3.7
Figure 3.5 CAG workflow: external knowledge is loaded once as a cached prompt prefix, which subsequent user queries reuse during generation
Follow the flow from external knowledge to a cached prefix reused by subsequent queries. section 3.7
KV-cache storage is itself a constraint: storing keys and values for past tokens grows linearly with context length and can consume substantial GPU memory. Serving-engine techniques such as quantization and offloading can reduce that footprint; some model layers require global information and are unsuitable for selective loading. section 12.3.2


