Attention weights relevant parts of a sequence more heavily than less relevant parts, helping models carry forward more useful information. It improved on recurrent neural networks, whose sequential processing limits training parallelism, but did not by itself resolve scalability. Self-attention relates positions within the sequence and lets Transformers process the input in parallel without recurrent structure. section 1.2
Operational implications for Transformer inference include:
- Attention layers can be computationally intensive because each next-token prediction is weighted by the relevance of previous tokens. section 4.3.1
- Prefill processes prompt tokens in parallel and populates the key–value cache; autoregressive sampling uses that state to decode subsequent tokens. section 4.3.1
- KV caching consumes GPU or CPU memory but avoids recomputing attention over all prior tokens, improving generation speed. section 4.3.1
- Arithmetic intensity separates memory-bound from compute-bound execution and helps identify whether attention performance is limited by data movement or compute capacity. section 4.3.1
Measuring KV-cache impact
A GPT-Neo generation test on a T4 GPU compares Transformers generation with use_cache enabled and disabled. The measured averages are 10.281 seconds with caching versus 51.978 seconds without it. section 4.3.1
Figure 4.1 The (Q × KT) × V computation process with caching
Shows the cached (Q × Kᵀ) × V attention computation and clarifies which state is reused during generation. section 4.3.1
