liveBook · Manning

WGMMA

WGMMA is an NVIDIA GPU instruction family that performs matrix multiply-accumulate operations collectively across a warp group.

How 2 Manning books explain it

CUDA for LLMs

WGMMA is an NVIDIA GPU instruction family that performs asynchronous matrix multiply-accumulate operations across a warp group.

How it is used
Accelerates large, compute-bound matrix multiplies on Hopper while overlapping tensor-core computation with tile movement.
How deep it goes
Walks through inline PTX operands, shared-memory descriptors, tiling, synchronization, and producer-consumer optimization, while deferring some library-grade microoptimizations.
Examples
Uses a 64 × 64 × 64 GEMM tile and an H100 throughput progression from basic WGMMA to pipelined kernels.
What you can do after
Build custom WGMMA kernels from inline PTX and evaluate tiling, synchronization, and memory-layout tradeoffs.

Read how CUDA for LLMs explains WGMMA →

Build a DeepSeek Model (From Scratch)

WGMMA is an NVIDIA GPU instruction that performs warp-group-level matrix multiply-accumulate operations.

How it is used
Accelerates FP8 matrix multiplication inside a precision-managed accumulation strategy.
How deep it goes
Explains Tensor Core execution, limited-precision accumulation, and periodic FP32 promotion, but not instruction syntax or programming code.
Examples
Uses y = Wx, FP8 activation and weight blocks, and diagrams of periodic partial-sum promotion.
What you can do after
Evaluate WGMMA's role in fast low-precision GEMM and explain why periodic FP32 accumulation improves numerical stability.

Read how Build a DeepSeek Model (From Scratch) explains WGMMA →

What to know first

Related concepts

Questions about WGMMA

What is WGMMA?

WGMMA is an NVIDIA GPU instruction family that performs matrix multiply-accumulate operations collectively across a warp group.

Which Manning books explain WGMMA?

CUDA for LLMs and Build a DeepSeek Model (From Scratch).

What should I know before learning WGMMA?

Manning books that teach WGMMA usually introduce bfloat16 and matrix-matrix multiplication first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.