chapter seven

7 Tensor cores

 

This chapter covers

  • Tensor core architecture and when to use tensor cores versus CUDA cores
  • WMMA fragments, tiling, and the high-level tensor core API (Volta and newer)
  • WGMMA asynchronous operations with inline PTX on Hopper
  • Tensor Memory Accelerator producer-consumer pipelines and circular buffering
  • Progressive optimization from 71 TFLOPS (WMMA) to 618 TFLOPS (WGMMA)
  • Mapping manual CUDA core patterns to their tensor core equivalents

Every major deep learning workload spends most of its GPU time performing matrix multiplications. Tensor cores exist to accelerate exactly this bottleneck.

A cluster of H100s trains a 70B-parameter model in weeks rather than months (Llama 3.1 70B took 7.0 million H100-hours).

Our best manually optimized GEMM from chapter 6 reaches 43 TFLOPS on an H100 (27.8 TFLOPS on the RTX 3090). By the end of this chapter, we will reach 618 TFLOPS, a 14× improvement. The jump comes from tensor cores, fixed-function matrix units that trade the flexibility of CUDA cores for massive throughput on a single operation: D = A × B + C.

7.1 Tensor core fundamentals

To use tensor cores effectively, we must understand their distinct architecture and execution model. Unlike general-purpose CUDA cores, tensor cores impose strict requirements on data layout and thread cooperation.

7.1.1 What are tensor cores?

7.1.2 When to use tensor cores

7.1.3 A brief history of tensor core instructions

7.1.4 Tensor core performance across GPU generations

7.2 WMMA: The high-level tensor core API

7.2.1 WMMA fragments and tiling strategy

7.2.2 Loading data with WMMA

7.2.3 Matrix multiply with WMMA

7.2.4 Storing results

7.3 WGMMA: Asynchronous warp group operations

7.3.1 Basic WGMMA: The first asynchronous tensor cores

7.3.2 Understanding PTX inline assembly

7.3.3 WGMMA with larger tiles

7.3.4 WGMMA with asynchronous memory operations