Prompt caching, also called prefix caching, reuses provider-side computation for repeated leading prompt content. Unlike response caching, it still makes an API call but lowers the cost of cached tokens, making it useful when large system prompts or document contexts remain stable. section 3.7.1
Treat caching as two complementary layers. Response caching returns identical results without contacting a provider; prompt caching reduces the price of requests that still reach one. A practical request path checks exact response cache first, then relies on provider caching for stable prompt prefixes. section 3.7.1
Implement provider caching behind the Model Service rather than exposing provider-specific behavior throughout applications:
- Pass provider-specific options through an adapter: Anthropic uses explicit cache-control markers, OpenAI caches automatically, and vLLM enables prefix caching by default. section 3.7.2
- Keep response-cache identity precise by hashing the model, messages, and configuration; temperature differences therefore produce distinct entries. section 3.7.2
- Make application caching configurable and bound it with TTL expiration and LRU eviction; enable or disable provider-level caching per request. section 3.7.2
- Record response-cache hits, provider-cached token counts, discounts, and the path taken so cost attribution distinguishes zero-cost hits from discounted and full-price calls. section 3.7.3 section 3.8.1
FAQ and document-assistant traffic
An FAQ service may serve 80% of requests from an exact response cache. For the remaining requests, a stable system prompt or repeated document context can receive provider prompt-caching discounts while the model still generates a fresh answer. section 3.7.1
Figure 3.7 The two levels of caching in LLM applications. Response caching (application level) provides the fastest, cheapest responses by avoiding API calls entirely. Prompt caching (provider level) reduces costs when the same prompt is reused but still requires API calls. Both strategies work together to optimize cost and performance.

Compare application-level response caching with provider-level prompt caching, including their API, latency, and cost trade-offs. section 3.7.1


