liveBook · Manning

Semantic caching

As explained in Build an Advanced RAG Application (From Scratch)

A semantic cache is a store of prior query results indexed so that semantically similar incoming queries can retrieve and reuse them.

Semantic caching stores prior query-answer results and reuses them for new queries with nearly the same meaning. In an Enterprise RAG pipeline, it avoids repeating routing, retrieval, and generation for recurring questions, reducing API cost, response latency, and pressure on rate limits. section 7.1 section 7.3

Exact-match caches miss natural-language paraphrases. A semantic cache instead embeds each query as a dense vector, finds the nearest cached query, and returns its answer only when similarity clears a threshold. Thus, “What was Uber's revenue in 2021?” and “How much money did Uber make in 2021?” can share a result despite different wording. section 7.3.1

A practical cache can use FAISS IndexFlatL2 for exact nearest-neighbor search over float32 query embeddings, with JSON holding the original queries, answers, and persisted embeddings. At thousands to tens of thousands of entries, exact search favors simple, precise behavior over approximate-index configuration. A conservative L2 threshold of 0.2 accepts close paraphrases while rejecting related but distinct requests such as different reporting years. section 7.3.2

Implement the cache as a small component with clear responsibilities:

  • Encode the incoming query once; return its embedding from lookup so a miss can reuse it during insertion. section 7.3.3
  • Keep lookup and insertion separate: check_cache() searches without mutation, while add_to_cache() stores completed results. section 7.3.3
  • Reload persisted embeddings into FAISS at startup; otherwise answers on disk remain unavailable to the rebuilt in-memory index. section 7.3.3
  • Bypass both lookup and storage for time-sensitive queries. A conservative keyword filter sends terms such as “latest,” “today,” or “outage” to fresh web search instead of risking stale answers. section 7.3.3

Hit versus miss

The first Uber revenue question runs the full RAG path and is cached. A later paraphrase becomes a cache hit and returns immediately, skipping routing, rewriting, retrieval, and generation. Typical cache hits use one embedding call and no LLM calls; full RAG misses require routing and generation calls. section 7.3.3 section 7.5

Figure 7.5 The core concept of semantic caching within an Enterprise RAG system to help reduce processing time to look up time for similar queries previously asked

The diagram contrasts expensive repeated processing with fast lookup for a previously answered similar query. section 7.3.2

Place the cache at the pipeline entrance. On a hit, return the stored answer; otherwise check time sensitivity, route the query, rewrite or decompose it when needed, retrieve, synthesize a grounded response, and cache the completed query-response pair. Returning processing metadata preserves an audit trail for debugging, compliance, and improvement. section 7.5

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Builds a FAISS-and-JSON cache through filtering, lookup, persistence, thresholds, and unified-pipeline integration; defers approximate indexes until larger scale.
Examples
Uses paraphrased Uber 2021 revenue questions to demonstrate a miss followed by a cache hit and measured speedup.
What it assumes
Introduces query embeddings, vector search, cache persistence, and time-sensitive bypasses through an incremental Python implementation.
What you can do after
Build it from scratch as a persisted FAISS semantic cache with conservative freshness handling inside an Enterprise RAG pipeline.
How it connects
Connects semantic caching with RAG, query routing, query rewriting, sub-query decomposition, embeddings, and live web search.

Also explained in

Compare how 3 books explain semantic caching →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.