6 Optimizing our transformer kernels
This chapter covers
- Profiling transformer kernels with Nsight Compute to pinpoint real bottlenecks
- Upgrading GEMV, softmax, LayerNorm, top-k, and GEMM through a repeatable optimization playbook
- Balancing memory bandwidth, register pressure, and occupancy as first-class design constraints
- Deciding when shared memory, vectorization, and warp-level intrinsics actually pay off
- Validating every improvement with progressive benchmarks and checks
- Packaging the resulting patterns so they transfer cleanly to production transformer stacks
The gap between “correct” and “fast” can span two orders of magnitude, and closing that gap requires understanding how GPUs actually move and process data. This chapter takes five kernels central to transformer inference: GEMV, softmax, LayerNorm, top-k, and GEMM, and optimizes each one from first principles. We will profile each kernel, identify its bottleneck, apply a targeted fix, and measure the result.
6.1 Preparing to optimize
Before we can effectively optimize our kernels, we need to establish a robust baseline and a reliable way to measure improvements. This section sets up the profiling tools and methods that will guide our optimization decisions throughout the chapter.
6.1.1 Profiling CUDA kernels
Before we start optimizing, we need to understand how to measure kernel performance. NVIDIA provides two primary profiling tools: