A key-value (KV) cache stores attention keys and values from earlier tokens for reuse during autoregressive inference. Without it, each decode step reprocesses the growing sequence. With it, the model computes projections for the new token and retrieves past keys and values from memory. section 2.3 section 2.4.1
A cached inference loop has two phases:
- Prefill: process the prompt in one forward pass and populate the cache. section 2.4.2
- Decode: project the new token into a query, key, and value; append its key and value to the cache; then use the new query with the full cached matrices to compute attention. section 2.4.2
- Repeat with the updated cache. In standard multi-head attention, caches are maintained separately for each layer and head. Past queries are not cached: only the current token's query is needed. section 2.4.2 section 2.4.1
Appending one token
If cached keys and values each have shape 4 × 4, projecting a fifth token adds one row to each matrix. The new query attends over the resulting 5 × 4 matrices; the earlier projections need not be recomputed. section 2.4.1
For attention scores per head and layer, caching reduces a decode step from O(n²) to O(n), where n is context length. Across generation, the corresponding attention-score cost changes from O(n³) to O(n²). The cache saves computation, but it must grow with the sequence. section 2.3.3 section 2.5
Measure the speedup
Generate 100 tokens with pretrained GPT-2 using Hugging Face's use_cache flag. One CPU-laptop run took 30.98 seconds without caching and 6.16 seconds with it, a 5.03× speedup. Hardware, sequence lengths, and warm-up affect the result. section 2.4.3
Listing 2.2 Demonstrating the Speedup of KV Caching
prompt = "The next day is bright"
inputs = tokenizer(prompt, return_tensors="pt")
input_ids = inputs.input_ids
attention_mask = inputs.attention_mask
# Timing without KV cache
start_time_without_cache = time.time()
output_without_cache = model.generate(input_ids,
[CA] max_new_tokens=100,
[CA] use_cache=False, #A
[CA] attention_mask=attention_mask)
end_time_without_cache = time.time()
duration_without_cache = end_time_without_cache - start_time_without_cache
print(f"Time without KV Cache: {duration_without_cache:.4f} seconds")
# Timing with KV cache
start_time_with_cache = time.time()
output_with_cache = model.generate(input_ids,
[CA] max_new_tokens=100,
[CA] use_cache=True, #B
[CA] attention_mask=attention_mask)
end_time_with_cache = time.time()
duration_with_cache = end_time_with_cache - start_time_with_cache
print(f"Time with KV Cache: {duration_with_cache:.4f} seconds")
# Calculate and print the speedup
speedup = duration_without_cache / duration_with_cache
print(f"\nKV Cache Speedup: {speedup:.2f}x")Compare the two generation calls and their use_cache settings. section 2.4.3
For standard multi-head attention, cache bytes = layers × batch size × heads × head size × sequence length × 2 × bytes per element. The factor of two stores both keys and values; FP16 or BF16 uses two bytes per element. Besides capacity, repeatedly reading the cache can make inference memory-bandwidth bound. section 2.5.1 section 2.5
Multi-query attention shares one key and value projection across heads; grouped-query attention shares them within groups, trading memory savings against head specialization. Multi-head latent attention instead caches a smaller latent representation and reconstructs keys and values for attention. section 2.9 section 3.1
With rotary positional encoding, cached decoding must account for the new token's position. A decoupled attention implementation carries cached content and position tensors, derives a position offset from cache length, and returns the updated cache for the next step. section 6.2.1 section 6.2.2
Listing 6.10 The forward method of DeepSeekAttention
def forward(self, x: torch.Tensor, attn_mask: torch.Tensor,
past_kv: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
position_offset: int = 0):
B, S, D = x.shape
past_len = past_kv[0].shape[1] if past_kv is not None else 0
# --- Path A: Content Path —
q_c = self.W_q_content(x).view(B, S, self.num_heads, self.d_head).transpose(1, 2)
c_kv_new = self.W_dkv_content(x) #A
c_kv = torch.cat([past_kv[0], c_kv_new], dim=1) if past_kv else c_kv_new #B
k_c = self.W_uk_content(c_kv).view(B, past_len + S, ...).transpose(1, 2)
v_c = self.W_uv_content(c_kv).view(B, past_len + S, ...).transpose(1, 2)
# --- Path B: Position Path —
k_r_unrotated = self.W_k_pos(x).view(B, S, self.num_heads, self.d_rope).transpose(1, 2)
q_r_unrotated = self.W_q_pos(x).view(B, S, self.num_heads, self.d_rope).transpose(1, 2)
k_r_new = self.rope(k_r_unrotated, position_offset=position_offset) #C
q_r = self.rope(q_r_unrotated, position_offset=position_offset)
k_r = torch.cat([past_kv[1], k_r_new], dim=2) if past_kv else k_r_new #D
# --- Combining Paths —
content_scores = (q_c @ k_c.transpose(-2, -1)) / math.sqrt(self.d_head)
position_scores = (q_r @ k_r.transpose(-2, -1)) / math.sqrt(self.d_rope)
attn_scores = content_scores + position_scores
# ... (softmax, dropout, output projection) ...
output = self.W_o(context_vector)
new_cache = (c_kv, k_r) #E
return output, new_cacheInspect how the attention forward method appends new latent vectors to past cached tensors. section 6.2.1








