chapter five
5 Transformer inference in CUDA
This chapter covers
- Understanding the decoder-only transformer architecture that powers GPT-style models
- The fundamental split between prefill (processing the prompt) and decode (generating tokens)
- Building naive CUDA kernels for inference: GEMV, softmax, layer normalization, and attention
- Key-value caching to avoid redundant computation during autoregressive generation
- Mixture-of-experts routing with custom top-k kernels
- The CUDA-Python binding pipeline that connects our kernels to PyTorch
Chapter 4 gave you the complete training story: forward passes, backpropagation, gradient updates. Those principles apply to any architecture. Inference, which generates text one token at a time, has a different profile. Batch sizes shrink to one. Matrix multiplications become matrix-vector products. Memory bandwidth, not compute throughput, becomes the bottleneck.
5.1 Getting started: Transformer architecture and setup
Before writing inference kernels, we need to understand what we’re optimizing. This section establishes the transformer architecture, the terminology we’ll use throughout, and the PyTorch baseline that our CUDA kernels must match numerically.