Cache-augmented generation optimizes retrieval-augmented generation by reusing information or computation to improve retrieval efficiency and reduce inference cost. It remains within the modular retrieval-and-generation architecture rather than replacing it. section 1.3.4
Caching belongs alongside query routing and resource management as an implementation choice. Its value is practical: avoid repeating expensive work while preserving access to externally retrieved information and grounded generation. section 1.3.4
Use the technique selectively:
- Prefer cached or preloaded context when the corpus is small, fixed, and fits within the model's context window; retrieval may add complexity without improving results. section 1.3.5
- Retain retrieval when the corpus or query workload makes external access valuable, and evaluate the crossover rather than assuming caching is always superior. section 1.3.5
Small fixed corpus
For a small, stable corpus that fits in context, load the material once and skip the retriever. Reported comparisons across 2,326 cases and eleven models found long context ahead by 2.4% at 32,000 tokens, while retrieval led by 3.68% at 128,000 tokens. section 1.3.5

