liveBook · Manning

Caching

As explained in LLMs in Production

Caching is the technique of storing reusable computation results or data so later requests can be served without repeating the original work.

Caching stores reusable results so repeated, easily normalized requests can be served without running the language model again. This can reduce latency because the fastest inference call is the one avoided. section 6.3.3

Use caching pragmatically:

  • Consider it when users commonly submit the same requests and their inputs can be normalized reliably. section 6.3.3
  • Treat it as one option among GPU optimization, gRPC, smaller models, load testing, and profiling when latency is poor. section 6.3.3 section 6.5
  • Measure before optimizing, because the slowest stage may be elsewhere in the inference pipeline. section 6.3.3

How LLMs in Production covers it

How deep it goes
Recommends caching conditionally but stops short of specifying cache keys, invalidation, storage, or implementation.
Examples
Uses repeated user requests with easily normalized inputs as the motivating example.
What it assumes
Assumes familiarity with software services, LLM inference, request inputs, latency, load testing, and profiling.
What you can do after
Apply caching when request repetition and normalization make avoided inference likely to improve service speed.
How it connects
Connects caching with latency, inference, input normalization, load testing, profiling, GPU optimization, and model selection.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

Compare how 38 books explain caching →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.