Semantic caching reuses an earlier LLM response when embeddings judge a new query equivalent or sufficiently similar in meaning, even when its wording differs. It targets predictable, repeated questions to reduce latency and token-based costs in production systems. section 13.4.3 section 13.5
Production designs must balance reuse against correctness and operating cost:
- Choose similarity thresholds carefully: values above 0.95 can miss valid matches, while values around 0.85 can return unsuitable answers. section 13.4.3
- Manage freshness through time relevance and semantic drift, because simple TTL policies may not address changing responses or language. section 13.4.3
- Account for embedding-lookup overhead; caching helps most when question patterns repeat frequently. section 13.4.3
Equivalent authentication questions
A cache can treat “How do I rotate API keys?” and “What’s the process for updating authentication credentials?” as similar queries and reuse the first response. section 13.4.3
Listing 13.15 Semantic caching with RedisVL vector similarity
class SemanticCache:
def __init__(self, similarity_threshold: float = 0.92):
self.threshold = similarity_threshold #1
self.embeddings = GeminiEmbeddings()
self.index = Index.from_yaml("cache_schema.yaml") #2
self.index.create()
async def get_cached_response(self, query: str) -> Optional[Dict]:
query_embedding = ( #3
self.embeddings.embed_query(query)) #3
vector_query = VectorQuery(
vector=query_embedding,
vector_field_name="query_embedding",
num_results=1, #4
return_fields=["cached_response", "original_query", "timestamp"]
)
results = self.index.query(vector_query)
if (results and #5
results[0]["vector_score"] >= self.threshold): #5
cache_hit = results[0]
return {
"response": json.loads(cache_hit["cached_response"]),
"similarity": cache_hit["vector_score"],
"original_query": cache_hit["original_query"]
}
return None #6
async def cache_response(self, query: str, response: Dict):
query_embedding = self.embeddings.embed_query(query)
cache_entry = {
"id": hashlib.md5(query.encode()).hexdigest(), #7
"original_query": query,
"query_embedding": query_embedding,
"cached_response": json.dumps(response), #8
"timestamp": datetime.now().isoformat()
}
self.index.load([cache_entry]) #9
Inspect a RedisVL vector-similarity implementation that embeds each query, searches the cache, and accepts the nearest result only when its score meets the configured threshold. section 13.4.3
