liveBook · Manning

Prompt caching

As explained in Designing AI Systems

Prefix caching is a caching method that reuses computations for an identical leading portion of a prompt.

Prompt caching, also called prefix caching, reuses provider-side computation for repeated leading prompt content. Unlike response caching, it still makes an API call but lowers the cost of cached tokens, making it useful when large system prompts or document contexts remain stable. section 3.7.1

Treat caching as two complementary layers. Response caching returns identical results without contacting a provider; prompt caching reduces the price of requests that still reach one. A practical request path checks exact response cache first, then relies on provider caching for stable prompt prefixes. section 3.7.1

Implement provider caching behind the Model Service rather than exposing provider-specific behavior throughout applications:

  • Pass provider-specific options through an adapter: Anthropic uses explicit cache-control markers, OpenAI caches automatically, and vLLM enables prefix caching by default. section 3.7.2
  • Keep response-cache identity precise by hashing the model, messages, and configuration; temperature differences therefore produce distinct entries. section 3.7.2
  • Make application caching configurable and bound it with TTL expiration and LRU eviction; enable or disable provider-level caching per request. section 3.7.2
  • Record response-cache hits, provider-cached token counts, discounts, and the path taken so cost attribution distinguishes zero-cost hits from discounted and full-price calls. section 3.7.3 section 3.8.1

FAQ and document-assistant traffic

An FAQ service may serve 80% of requests from an exact response cache. For the remaining requests, a stable system prompt or repeated document context can receive provider prompt-caching discounts while the model still generates a fresh answer. section 3.7.1

Figure 3.7 The two levels of caching in LLM applications. Response caching (application level) provides the fastest, cheapest responses by avoiding API calls entirely. Prompt caching (provider level) reduces costs when the same prompt is reused but still requires API calls. Both strategies work together to optimize cost and performance.

Compare application-level response caching with provider-level prompt caching, including their API, latency, and cost trade-offs. section 3.7.1

How Designing AI Systems covers it

How deep it goes
Explains caching layers, provider adapters, cache controls, keying, eviction, and monitoring, but not provider cache internals.
Examples
Uses FAQ systems, stable system prompts, repeated document context, Anthropic markers, OpenAI automatic caching, and vLLM prefix caching.
What you can do after
Operate/configure provider caching through adapters, request options, cache policies, and cost-effectiveness metrics.
How it connects
Connects prompt caching with response caching, semantic caching, RAG, provider adapters, and observability.

Also explained in

Compare how 3 books explain prompt caching →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.