liveBook · Manning

Prompt caching

As explained in LLM Customization and Fine-Tuning

A technique that stores reusable prompt prefixes or inputs so repeated content does not need to be processed or billed in the same way each time.

Prompt caching reduces the cost of repeated prompt prefixes—such as system prompts and few-shot examples—when using a managed API. Savings vary by provider and matter most when repeated input is a substantial share of the prompt. section 2.4.1

Check what repeats before estimating the savings:

  • A repeated 3,000-token demonstration block can roughly halve the bill; short prompts save less when output tokens dominate. section 2.4.1
  • Retrieved chunks change with each query, so prompt caching does not discount them. section 2.4.1
Figure 2.2 Managed-API cost scales with request volume; self-hosted cost is largely fixed.

Use the cost comparison to weigh managed-API discounts, including caching, against largely fixed self-hosting costs. section 2.4.1

How LLM Customization and Fine-Tuning covers it

How deep it goes
Quantifies provider discounts and explains which prompt content benefits, without detailing cache implementation.
Examples
Uses a 3,000-token demonstration block and contrasts reusable prefixes with changing retrieved chunks.
What you can do after
Evaluate whether repeated prompt prefixes can materially reduce managed-API costs before comparing self-hosting.
How it connects
Connects to managed APIs, few-shot prompting, retrieval, batch APIs, and self-hosting cost comparisons.

Also explained in

Compare how 3 books explain prompt caching →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.