Response caching stores complete model responses and reuses them when a later request matches. A hit avoids the provider call, reducing cost and latency. It suits repeated, deterministic queries, but it defeats applications that need fresh variations for the same input. section 3.7.1 section 3.10
Exact-match caching requires the same messages and configuration. Semantic caching instead embeds queries and returns a cached response when similarity exceeds a configured threshold. It can improve hit rates for differently worded questions, at the cost of precision. The strategies can be layered: check exact matches first, then semantic similarity on a miss. section 3.7.1
Repeated support questions
A documentation bot can reuse an answer to “How do I authenticate?” when the same request recurs. A semantic cache may also match differently worded password-reset questions, while a creative-writing assistant may need a new response instead. section 3.7.1
A response-cache interface can accept a model, messages, and chat configuration for both lookup and storage. Hashing those inputs distinguishes requests with different temperatures. Configure caching per request, expire stale entries with a TTL, and evict least-recently-used entries when capacity is reached. section 3.7.2
Listing 3.15 CacheConfig and ResponseCache interface
@dataclass
class CacheConfig:
enabled: bool = True
ttl_seconds: int = 3600 #A
max_size: int = 10000
class ResponseCache:
def get(self, model: str, messages: List[ChatMessage],
config: ChatConfig) -> Optional[ChatResponse] #B
def set(self, model: str, messages: List[ChatMessage],
config: ChatConfig, response: ChatResponse)
The interface specifies cache configuration and the inputs used to retrieve or store a response. section 3.7.2
Prompt caching is complementary: the provider reuses portions of repeated prompts, but the application still makes an API call and pays a discounted rate for cached tokens. Track response-cache hit rate and prompt-cache token savings separately, then attribute costs to the path each request took. section 3.7.1 section 3.7.3
Figure 3.7 The two levels of caching in LLM applications. Response caching (application level) provides the fastest, cheapest responses by avoiding API calls entirely. Prompt caching (provider level) reduces costs when the same prompt is reused but still requires API calls. Both strategies work together to optimize cost and performance.

The diagram contrasts application-level response reuse with provider-level prompt reuse. section 3.7.1
















































