A key-value (KV) cache stores the intermediate attention keys and values computed for previously processed tokens. During autoregressive generation, the transformer reuses those states instead of recomputing attention states for the existing prefix. section 3.7
A shared context can be encoded once and reused for multiple generations. For each query, generation starts with the shared context’s cached state, processes the query, and then passes the updated cache back to the model as it generates successive tokens. section 3.10.7
Listing 3.8 Minimal KV-cache optimization script using GPT-2 and PyTorch
import torch
inputs = tokenizer(context, return_tensors="pt").to(device) #1
with torch.no_grad():
outputs = model(**inputs, use_cache=True)
kv_cache = outputs.past_key_values
input_ids = tokenizer(query, return_tensors="pt").to(device).input_ids #2
current_cache = kv_cache
generated_token_ids = []
for _ in range(max_new_tokens):
with torch.no_grad():
outputs = model(
input_ids=input_ids,
past_key_values=current_cache,
use_cache=True
)
next_token_id = torch.argmax(outputs.logits[:,-1,:], dim=-1).
unsqueeze(-1)
generated_token_ids.append(next_token_id.item())
if next_token_id.item() == tokenizer.eos_token_id:
break
current_cache = outputs.past_key_values
input_ids = next_token_id
answer = tokenizer.decode(generated_token_ids, skip_special_tokens=True)
The GPT-2 example builds a cache from a context, passes it as past_key_values with a query, and updates it during token generation. section 3.10.7
The KV cache is an internal inference optimization. Prompt caching exposes reuse of identical static prefixes through a provider API, while cache-augmented generation (CAG) adds application decisions about what knowledge to preload, retain, and invalidate. A CAG system must still work on a cache miss; a hit changes latency and cost, not prompt content. section 3.7 section 3.10.7
Cache memory is a major long-context inference bottleneck: storing keys and values for past tokens grows linearly with context length. Serving engines can manage the cache lifecycle across requests; TailorKV combines quantization and offloading to reduce its footprint, while accounting for layers that need global information. section 12.3.2








