liveBook · Manning

Key-value cache

As explained in Context Engineering

A key-value cache stores previously computed transformer attention keys and values for reuse during generation.

A key-value (KV) cache stores the intermediate attention keys and values computed for previously processed tokens. During autoregressive generation, the transformer reuses those states instead of recomputing attention states for the existing prefix. section 3.7

A shared context can be encoded once and reused for multiple generations. For each query, generation starts with the shared context’s cached state, processes the query, and then passes the updated cache back to the model as it generates successive tokens. section 3.10.7

Listing 3.8 Minimal KV-cache optimization script using GPT-2 and PyTorch
import torch

inputs = tokenizer(context, return_tensors="pt").to(device) #1
with torch.no_grad():
    outputs = model(**inputs, use_cache=True)

kv_cache = outputs.past_key_values

input_ids = tokenizer(query, return_tensors="pt").to(device).input_ids #2
current_cache = kv_cache
generated_token_ids = []
for _ in range(max_new_tokens):
    with torch.no_grad():
        outputs = model(
            input_ids=input_ids,
            past_key_values=current_cache,
            use_cache=True
        )
    next_token_id = torch.argmax(outputs.logits[:,-1,:], dim=-1).
    unsqueeze(-1)
    generated_token_ids.append(next_token_id.item())
    if next_token_id.item() == tokenizer.eos_token_id:
        break
    current_cache = outputs.past_key_values
    input_ids = next_token_id

answer = tokenizer.decode(generated_token_ids, skip_special_tokens=True)

The GPT-2 example builds a cache from a context, passes it as past_key_values with a query, and updates it during token generation. section 3.10.7

The KV cache is an internal inference optimization. Prompt caching exposes reuse of identical static prefixes through a provider API, while cache-augmented generation (CAG) adds application decisions about what knowledge to preload, retain, and invalidate. A CAG system must still work on a cache miss; a hit changes latency and cost, not prompt content. section 3.7 section 3.10.7

Cache memory is a major long-context inference bottleneck: storing keys and values for past tokens grows linearly with context length. Serving engines can manage the cache lifecycle across requests; TailorKV combines quantization and offloading to reduce its footprint, while accounting for layers that need global information. section 12.3.2

How Context Engineering covers it

How deep it goes
Demonstrates cache creation and token-by-token reuse, then discusses memory growth and serving-engine optimizations.
Examples
Uses a shared context and query with GPT-2 to demonstrate cached generation.
What it assumes
Assumes an initialized model, tokenizer, device, context, query, and token limit; omits their setup from the example.
What you can do after
Apply it via PyTorch and Hugging Face Transformers using past_key_values during generation.
How it connects
Connects attention state to prompt caching, cache-augmented generation, quantization, and offloading.

Also explained in

Compare how 9 books explain key-value cache →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Context Engineering. The section links point to the parts of the book it draws on.