Caching stores generated answers so repeated queries do not require another LLM call. For a busy support bot, that can reduce response time and API costs. Exact-match caching is simple, but it misses differently worded questions that need the same answer. section 9.2.3
Two approaches differ in what counts as a hit:
- Exact matching looks up the query string directly. It works well for verbatim repeats. section 9.2.3
- Semantic caching compares query embeddings, allowing similar wording to retrieve a stored response. A similarity threshold controls whether to reuse it or call the LLM. section 9.2.3
def handle_query(query): #1
cached_response = get_cached_response(query) #2
if cached_response:
return cached_response
response = call_gpt_api(query) #3
vector_store.add_texts([query], metadatas=[{"response": response}]) #4
return response
def call_gpt_api(query): #5
#6
print(handle_query("What’s your refund policy?")) #7
The query handler checks the cache first, calls the LLM on a miss, then stores the new query-response pair for reuse. section 9.2.3
