Semantic caching reuses an LLM response when a new query has a similar meaning to a cached query. Unlike an exact-string cache, it can match differently worded questions. For a busy customer-support service, that can avoid repeated model calls, reducing latency and cost. section 9.2.3
Store-hours questions
“What are your hours?” and “Tell me your operating hours” can retrieve the same stored answer despite their different wording. An exact-string lookup would treat them as separate queries. section 9.2.3
A cache-backed request follows four steps:
- Convert the query to a vector embedding and search stored query embeddings for a similar match. section 9.2.3
- If a match passes the configured similarity threshold, return its stored response. section 9.2.3
- Otherwise, call the LLM and store the new query with its response for later reuse. section 9.2.3
def handle_query(query): #1
cached_response = get_cached_response(query) #2
if cached_response:
return cached_response
response = call_gpt_api(query) #3
vector_store.add_texts([query], metadatas=[{"response": response}]) #4
return response
def call_gpt_api(query): #5
#6
print(handle_query("What’s your refund policy?")) #7
The request handler shows the cache-first path: look up a response, call the model on a miss, then save that response with the query. section 9.2.3
One implementation stores responses as metadata alongside queries in FAISS and uses embeddings for retrieval. Another wraps model calls with GPTCache, an ONNX embedding model, and a 0.75 similarity threshold. The threshold controls which near-matches are reused; similarity alone does not establish that a cached answer is appropriate for every variation of a question. section 9.2.3
In an e-commerce service, caching sits alongside request batching and model routing to handle traffic spikes without spending on repeated FAQ answers. The illustrated project’s mock-response mode bypasses the GPTCache-backed call, while its streaming path uses a separate model call; test the live, non-streaming path when evaluating cache behavior. section 9.2.5
