Caching stores and reuses expensive LLM processing so production systems reduce repeated work, token consumption, and operational cost without changing response quality. section 13.4.3
Choose the strategy according to how often content and questions repeat:
- Context caching stores processed document chunks and reuses them across questions about the same documentation. It suits stable, frequently accessed knowledge bases such as internal documentation and FAQ systems. section 13.4.3
- Semantic caching uses embeddings to recognize differently worded questions that are fundamentally similar, rather than requiring exact query matches. section 13.4.3
- Client-side caching keeps remotely managed prompts local after the first fetch, while asynchronous refresh, fallback prompts, and prefetching protect latency and availability. section 13.1.3
Treat invalidation as an operational requirement: documentation updates can make cached content stale, while memory and storage costs can outweigh savings when queries rarely repeat. Dynamic or personalized content usually provides limited reuse. section 13.4.3
Embedding-backed semantic lookup
A RedisVL cache can embed an incoming query, search an index by vector similarity, return the cached response and metadata, and accept a hit only when the similarity score reaches a configured threshold such as 0.92. section 13.4.3
Listing 13.15 Semantic caching with RedisVL vector similarity
class SemanticCache:
def __init__(self, similarity_threshold: float = 0.92):
self.threshold = similarity_threshold #1
self.embeddings = GeminiEmbeddings()
self.index = Index.from_yaml("cache_schema.yaml") #2
self.index.create()
async def get_cached_response(self, query: str) -> Optional[Dict]:
query_embedding = ( #3
self.embeddings.embed_query(query)) #3
vector_query = VectorQuery(
vector=query_embedding,
vector_field_name="query_embedding",
num_results=1, #4
return_fields=["cached_response", "original_query", "timestamp"]
)
results = self.index.query(vector_query)
if (results and #5
results[0]["vector_score"] >= self.threshold): #5
cache_hit = results[0]
return {
"response": json.loads(cache_hit["cached_response"]),
"similarity": cache_hit["vector_score"],
"original_query": cache_hit["original_query"]
}
return None #6
async def cache_response(self, query: str, response: Dict):
query_embedding = self.embeddings.embed_query(query)
cache_entry = {
"id": hashlib.md5(query.encode()).hexdigest(), #7
"original_query": query,
"query_embedding": query_embedding,
"cached_response": json.dumps(response), #8
"timestamp": datetime.now().isoformat()
}
self.index.load([cache_entry]) #9
Inspect the RedisVL implementation showing threshold configuration, Gemini embeddings, index creation, vector queries, returned cache fields, and similarity-based hit selection. section 13.4.3
