liveBook · Manning

Cache-augmented generation

As explained in Retrieval Augmented Generation, The Foundational Ideas

A generation approach that augments retrieval with cached information or computation to improve efficiency and reduce inference cost.

Cache-augmented generation optimizes retrieval-augmented generation by reusing information or computation to improve retrieval efficiency and reduce inference cost. It remains within the modular retrieval-and-generation architecture rather than replacing it. section 1.3.4

Caching belongs alongside query routing and resource management as an implementation choice. Its value is practical: avoid repeating expensive work while preserving access to externally retrieved information and grounded generation. section 1.3.4

Use the technique selectively:

  • Prefer cached or preloaded context when the corpus is small, fixed, and fits within the model's context window; retrieval may add complexity without improving results. section 1.3.5
  • Retain retrieval when the corpus or query workload makes external access valuable, and evaluate the crossover rather than assuming caching is always superior. section 1.3.5

Small fixed corpus

For a small, stable corpus that fits in context, load the material once and skip the retriever. Reported comparisons across 2,326 cases and eleven models found long context ahead by 2.4% at 32,000 tokens, while retrieval led by 3.68% at 128,000 tokens. section 1.3.5

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Positions caching as an architectural implementation detail and selection aid, without specifying cache algorithms or operational procedures.
Examples
Uses small fixed corpora and compares long-context with retrieval performance across 2,326 cases and eleven models.
What it assumes
Assumes retrieval-augmented generation and distinguishes retrieval from generation within that architecture.
What you can do after
Evaluate whether a fixed small corpus should bypass retrieval and use cached or loaded context instead.
How it connects
Connects retrieval-augmented generation with query routing, resource management, fine-tuning, and long-context processing.

Also explained in

Compare how 2 books explain cache-augmented generation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.