Semantic caching
Semantic caching is a technique that stores and reuses results for requests judged sufficiently similar in meaning, rather than requiring identical text or keys.
Building Reliable AI Systems
A caching technique that reuses results for queries with similar meanings, rather than requiring exact textual matches.
- How it is used
- Reuses answers to repeated or similarly phrased support queries, reducing redundant LLM calls during heavy traffic.
- How deep it goes
- Works through embedding storage, similarity lookup, thresholded reuse, and cache-miss handling, with FAISS and GPTCache implementation sketches.
- Examples
- Uses differently phrased store-hours questions and an e-commerce support service with repeated refund queries.
- What you can do after
- Apply it via FAISS or GPTCache to return cached responses on similar-query hits and call the LLM on misses.
Read how Building Reliable AI Systems explains semantic caching →
Machine Learning Platform Engineering
Semantic caching is a caching method that reuses results for inputs judged equivalent or sufficiently similar in meaning rather than identical in text.
- How it is used
- Reduces repeated LLM calls, latency, and token costs for predictable question patterns in production systems.
- How deep it goes
- Explains embeddings, threshold trade-offs, freshness, semantic drift, and lookup overhead, then demonstrates vector-similarity implementation.
- Examples
- Uses paraphrased API-key questions and a RedisVL vector-similarity cache with configurable thresholds.
- What you can do after
- Build semantic lookup with embeddings and RedisVL, while tuning thresholds and managing response freshness.
Read how Machine Learning Platform Engineering explains semantic caching →
Related concepts
- embeddinguses · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
- FAISSuses · FAISS is an open-source software library for efficient similarity search and clustering of dense vectors.
- GPTCacheuses · GPTCache is an open-source library that caches large-language-model responses and can reuse semantically similar results.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.