liveBook · Manning

WGMMA

As explained in Build a DeepSeek Model (From Scratch)

WGMMA is an NVIDIA GPU instruction that performs warp-group-level matrix multiply-accumulate operations.

WGMMA (Warp-Group-level Matrix Multiply-Accumulate) is a Tensor Core instruction that performs fast, low-precision matrix-multiply bursts for operations such as y = Wx. section 5.6.3

Its role in precision-managed accumulation is defined by three architectural steps:

  • Consumes FP8 activation and weight tensors as GEMM inputs. section 5.6.3
  • Accumulates intermediate products in a limited-precision Tensor Core register. section 5.6.3
  • At a fixed interval Nc, promotes partial sums to CUDA Cores for final accumulation in an FP32 register, reducing underflow risk. section 5.6.3
Y_ij = W_i1*X_1j + W_i2*X_2j + ... + W_ik*X_kj

Use the matrix equation to follow how WGMMA contributes products to each output element. section 5.6.3

Figure 5.31 The Increasing Accumulation Precision mechanism in detail. Low-precision accumulation happens in bursts on the Tensor Core, and results are periodically promoted to a high-precision FP32 register on the CUDA Core.

Trace the alternating fast low-precision Tensor Core bursts and periodic high-precision CUDA Core accumulation. section 5.6.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains Tensor Core execution, limited-precision accumulation, and periodic FP32 promotion, but not instruction syntax or programming code.
Examples
Uses y = Wx, FP8 activation and weight blocks, and diagrams of periodic partial-sum promotion.
What it assumes
Assumes matrix multiplication and FP8 tensors, while grounding the distinction between Tensor Cores and CUDA Cores.
What you can do after
Evaluate WGMMA's role in fast low-precision GEMM and explain why periodic FP32 accumulation improves numerical stability.
How it connects
Ties WGMMA to Tensor Cores, CUDA Cores, FP8, FP32 accumulation, GEMM, and numerical underflow.

Also explained in

Compare how 2 books explain WGMMA →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.