7 Tensor cores
This chapter covers
- Tensor core architecture and when to use tensor cores versus CUDA cores
- WMMA fragments, tiling, and the high-level tensor core API (Volta and newer)
- WGMMA asynchronous operations with inline PTX on Hopper
- Tensor Memory Accelerator producer-consumer pipelines and circular buffering
- Progressive optimization from 71 TFLOPS (WMMA) to 618 TFLOPS (WGMMA)
- Mapping manual CUDA core patterns to their tensor core equivalents
Every major deep learning workload spends most of its GPU time performing matrix multiplications. Tensor cores exist to accelerate exactly this bottleneck.
A cluster of H100s trains a 70B-parameter model in weeks rather than months (Llama 3.1 70B took 7.0 million H100-hours).
Our best manually optimized GEMM from chapter 6 reaches 43 TFLOPS on an H100 (27.8 TFLOPS on the RTX 3090). By the end of this chapter, we will reach 618 TFLOPS, a 14× improvement. The jump comes from tensor cores, fixed-function matrix units that trade the flexibility of CUDA cores for massive throughput on a single operation: D = A × B + C.
7.1 Tensor core fundamentals
To use tensor cores effectively, we must understand their distinct architecture and execution model. Unlike general-purpose CUDA cores, tensor cores impose strict requirements on data layout and thread cooperation.