Semantic caching stores prior query-answer results and reuses them for new queries with nearly the same meaning. In an Enterprise RAG pipeline, it avoids repeating routing, retrieval, and generation for recurring questions, reducing API cost, response latency, and pressure on rate limits. section 7.1 section 7.3
Exact-match caches miss natural-language paraphrases. A semantic cache instead embeds each query as a dense vector, finds the nearest cached query, and returns its answer only when similarity clears a threshold. Thus, “What was Uber's revenue in 2021?” and “How much money did Uber make in 2021?” can share a result despite different wording. section 7.3.1
A practical cache can use FAISS IndexFlatL2 for exact nearest-neighbor search over float32 query embeddings, with JSON holding the original queries, answers, and persisted embeddings. At thousands to tens of thousands of entries, exact search favors simple, precise behavior over approximate-index configuration. A conservative L2 threshold of 0.2 accepts close paraphrases while rejecting related but distinct requests such as different reporting years. section 7.3.2
Implement the cache as a small component with clear responsibilities:
- Encode the incoming query once; return its embedding from lookup so a miss can reuse it during insertion. section 7.3.3
- Keep lookup and insertion separate: check_cache() searches without mutation, while add_to_cache() stores completed results. section 7.3.3
- Reload persisted embeddings into FAISS at startup; otherwise answers on disk remain unavailable to the rebuilt in-memory index. section 7.3.3
- Bypass both lookup and storage for time-sensitive queries. A conservative keyword filter sends terms such as “latest,” “today,” or “outage” to fresh web search instead of risking stale answers. section 7.3.3
Hit versus miss
The first Uber revenue question runs the full RAG path and is cached. A later paraphrase becomes a cache hit and returns immediately, skipping routing, rewriting, retrieval, and generation. Typical cache hits use one embedding call and no LLM calls; full RAG misses require routing and generation calls. section 7.3.3 section 7.5
Figure 7.5 The core concept of semantic caching within an Enterprise RAG system to help reduce processing time to look up time for similar queries previously asked
The diagram contrasts expensive repeated processing with fast lookup for a previously answered similar query. section 7.3.2
Place the cache at the pipeline entrance. On a hit, return the stored answer; otherwise check time sensitivity, route the query, rewrite or decompose it when needed, retrieve, synthesize a grounded response, and cache the completed query-response pair. Returning processing metadata preserves an audit trail for debugging, compliance, and improvement. section 7.5


