A production GenAI application may receive the same question many times. Caching stores reusable data or responses in faster-access storage so a repeat request need not trigger another LLM API call. For static or rarely changing information, this can reduce latency, token use, API fees, and pressure on rate limits. section 3.4.5 section 11.5.1 section 11.5.3
Use a cache-aside pattern, then make correctness and availability part of the design:
- Check the cache first. On a miss, request the response from its source, store it, and return it. section 11.5.3
- Make cache keys represent the request and its context. Set an invalidation policy, such as a time-to-live (TTL), to avoid serving outdated information. section 11.5.3
- Monitor hit rates and performance. Handle misses gracefully, and keep the application working when the cache is temporarily unavailable. section 11.5.3
Caching LLM responses with Redis
A Python example uses RedisVL’s SemanticCache alongside an Azure OpenAI chat-completion call. The cache connects to Redis at redis://localhost:6379 and sets a distance threshold of 0.1. Its sample questions include country capitals; the reported run shows cache hits and misses, with average times of about 0.765 seconds without caching and 0.234 seconds with caching. Those measurements illustrate this run, not a guaranteed speedup. section 11.5.3
Choose what to cache for a clear use case rather than caching every generation. Frequently repeated prompts are one candidate; embeddings stored with vector search are another. In an enterprise application, caching also sits alongside response filtering, monitoring, and security—not in place of them. section 3.4.5 section 11.5.3 section 10.2.3
