chapter five

5 Transformer inference in CUDA

 

This chapter covers

  • Understanding the decoder-only transformer architecture that powers GPT-style models
  • The fundamental split between prefill (processing the prompt) and decode (generating tokens)
  • Building naive CUDA kernels for inference: GEMV, softmax, layer normalization, and attention
  • Key-value caching to avoid redundant computation during autoregressive generation
  • Mixture-of-experts routing with custom top-k kernels
  • The CUDA-Python binding pipeline that connects our kernels to PyTorch

Chapter 4 gave you the complete training story: forward passes, backpropagation, gradient updates. Those principles apply to any architecture. Inference, which generates text one token at a time, has a different profile. Batch sizes shrink to one. Matrix multiplications become matrix-vector products. Memory bandwidth, not compute throughput, becomes the bottleneck.

5.1 Getting started: Transformer architecture and setup

Before writing inference kernels, we need to understand what we’re optimizing. This section establishes the transformer architecture, the terminology we’ll use throughout, and the PyTorch baseline that our CUDA kernels must match numerically.

5.1.1 Establishing the baseline scenario

5.2 Inference: Theory and optimization

5.2.1 Optimizing autoregressive generation

5.2.2 Deploying the optimized stack

5.3 Inference kernels

5.3.1 Memory layout and contiguity

5.3.2 GEMV: The inference workhorse

Summary