A speedup compares baseline execution time with optimized execution time. In inference optimization, it measures how scheduling, memory access, caching, and GPU utilization reduce latency without changing model architecture. section 9.3
Interpret speedup through the optimization being compared:
- KV caching avoids recomputing the full attention context during autoregressive generation by storing and reusing key and value vectors. section 9.3
- Tiling, shared memory, coalesced access, operation fusion, and CUDA thread hierarchies improve GPU execution, with optimized kernels reported at 2x to 4x over naïve kernels. section 9.4
Measuring cached inference
A Hugging Face Transformers benchmark runs Llama 3.2 3B Instruct with use_cache enabled and disabled, records generation durations, and compares the resulting latency. section 9.3
Figure 9.4 At each step, the query is computed for the new token, while past key and value vectors are cached and reused. This avoids recomputation of the entire attention context, dramatically reducing inference time.
The figure traces autoregressive generation: each new token computes a query while reusing previously cached key and value vectors. section 9.3
