liveBook · Manning

Caching

As explained in Designing AI Systems

Response caching is the storage and reuse of complete model responses for requests that match according to a defined request-matching rule.

Response caching stores complete model responses and reuses them when a later request matches. A hit avoids the provider call, reducing cost and latency. It suits repeated, deterministic queries, but it defeats applications that need fresh variations for the same input. section 3.7.1 section 3.10

Exact-match caching requires the same messages and configuration. Semantic caching instead embeds queries and returns a cached response when similarity exceeds a configured threshold. It can improve hit rates for differently worded questions, at the cost of precision. The strategies can be layered: check exact matches first, then semantic similarity on a miss. section 3.7.1

Repeated support questions

A documentation bot can reuse an answer to “How do I authenticate?” when the same request recurs. A semantic cache may also match differently worded password-reset questions, while a creative-writing assistant may need a new response instead. section 3.7.1

A response-cache interface can accept a model, messages, and chat configuration for both lookup and storage. Hashing those inputs distinguishes requests with different temperatures. Configure caching per request, expire stale entries with a TTL, and evict least-recently-used entries when capacity is reached. section 3.7.2

Listing 3.15 CacheConfig and ResponseCache interface
@dataclass
class CacheConfig:
    enabled: bool = True
    ttl_seconds: int = 3600    #A
    max_size: int = 10000
 
class ResponseCache:
    def get(self, model: str, messages: List[ChatMessage],
            config: ChatConfig) -> Optional[ChatResponse]    #B
    def set(self, model: str, messages: List[ChatMessage],
            config: ChatConfig, response: ChatResponse)
 

The interface specifies cache configuration and the inputs used to retrieve or store a response. section 3.7.2

Prompt caching is complementary: the provider reuses portions of repeated prompts, but the application still makes an API call and pays a discounted rate for cached tokens. Track response-cache hit rate and prompt-cache token savings separately, then attribute costs to the path each request took. section 3.7.1 section 3.7.3

Figure 3.7 The two levels of caching in LLM applications. Response caching (application level) provides the fastest, cheapest responses by avoiding API calls entirely. Prompt caching (provider level) reduces costs when the same prompt is reused but still requires API calls. Both strategies work together to optimize cost and performance.

The diagram contrasts application-level response reuse with provider-level prompt reuse. section 3.7.1

How Designing AI Systems covers it

How deep it goes
Defines cache keys, configuration, eviction, and monitoring; describes semantic matching without implementing it.
Examples
Contrasts repeated documentation and password-reset questions with a creative-writing assistant that needs varied responses.
What you can do after
Operate and configure response caching using per-request settings, TTL, capacity, and hit-rate monitoring.
How it connects
Connects response caching to semantic similarity, prompt caching, observability, and cost attribution.

Also explained in

Compare how 49 books explain caching →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.