Prompt caching
Prompt caching stores and reuses previously processed prompt content or computations to reduce repeated processing, cost, or latency.
Designing AI Systems
Prefix caching is a caching method that reuses computations for an identical leading portion of a prompt.
- How it is used
- Reduces model-serving cost for repeated prompt prefixes after application-level response caching misses.
- How deep it goes
- Explains caching layers, provider adapters, cache controls, keying, eviction, and monitoring, but not provider cache internals.
- Examples
- Uses FAQ systems, stable system prompts, repeated document context, Anthropic markers, OpenAI automatic caching, and vLLM prefix caching.
- What you can do after
- Operate/configure provider caching through adapters, request options, cache policies, and cost-effectiveness metrics.
LLM Customization and Fine-Tuning
A technique that stores reusable prompt prefixes or inputs so repeated content does not need to be processed or billed in the same way each time.
- How it is used
- Uses repeated-prefix discounts to refine managed-API costs before comparing them with self-hosting or adaptation.
- How deep it goes
- Quantifies provider discounts and explains which prompt content benefits, without detailing cache implementation.
- Examples
- Uses a 3,000-token demonstration block and contrasts reusable prefixes with changing retrieved chunks.
- What you can do after
- Evaluate whether repeated prompt prefixes can materially reduce managed-API costs before comparing self-hosting.
Read how LLM Customization and Fine-Tuning explains prompt caching →
Context Engineering
Prompt caching reuses previously computed representations of prompt context, particularly shared prefixes, to avoid processing them again.
- How it is used
- Enables reuse of shared knowledge prefixes in cache-augmented generation to reduce request processing cost and latency.
- How deep it goes
- Explains prefix reuse, cache expiry and miss handling, alongside serving-engine memory optimizations, without provider-specific implementation details.
- Examples
- Illustrates warming an external-knowledge prefix and appending subsequent user queries for fresh completions.
- What you can do after
- Evaluate cache reuse and plan for expiry and misses when managing shared knowledge prefixes.
Also covered in
- AI for Everyday IT
- Vibe EngineeringIn early access (MEAP)
- AI Agents in Action, Second Edition
- CUDA for LLMs
What to know first
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
- costCost is the monetary expenditure or other resource burden incurred to acquire, produce, operate, or perform something.
- expiryExpiry is the point in time or elapsed duration after which an item or resource is no longer considered valid, current, or available.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
Related concepts
- key-value cacheuses · A key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.
- promptuses · A prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
- cache-augmented generationused by · A generation technique that reuses cached information, context, or computation to reduce repeated retrieval or inference costs.
- Clineused by · Cline is an AI coding assistant available as a Visual Studio Code extension that can help plan and modify software projects and interact with development tools.
- cost reductionproduces · Cost reduction is a decrease in expenditure or resources required relative to a prior or alternative baseline.
- cachingkind of · Caching is the temporary storage of data or computed results for reuse, reducing the work or delay of retrieving or computing them again.
Questions about prompt caching
What is prompt caching?
Prompt caching stores and reuses previously processed prompt content or computations to reduce repeated processing, cost, or latency.
Which Manning books explain prompt caching?
Designing AI Systems, LLM Customization and Fine-Tuning and Context Engineering.
What should I know before learning prompt caching?
Manning books that teach prompt caching usually introduce token, cost, expiry and language model first.
What is prompt caching a kind of?
Prompt caching is a kind of caching. Caching is the temporary storage of data or computed results for reuse, reducing the work or delay of retrieving or computing them again.
What is prompt caching also called?
Prompt caching is also called prefix caching.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


