WGMMA
WGMMA is an NVIDIA GPU instruction family that performs matrix multiply-accumulate operations collectively across a warp group.
CUDA for LLMs
WGMMA is an NVIDIA GPU instruction family that performs asynchronous matrix multiply-accumulate operations across a warp group.
- How it is used
- Accelerates large, compute-bound matrix multiplies on Hopper while overlapping tensor-core computation with tile movement.
- How deep it goes
- Walks through inline PTX operands, shared-memory descriptors, tiling, synchronization, and producer-consumer optimization, while deferring some library-grade microoptimizations.
- Examples
- Uses a 64 × 64 × 64 GEMM tile and an H100 throughput progression from basic WGMMA to pipelined kernels.
- What you can do after
- Build custom WGMMA kernels from inline PTX and evaluate tiling, synchronization, and memory-layout tradeoffs.
Build a DeepSeek Model (From Scratch)
WGMMA is an NVIDIA GPU instruction that performs warp-group-level matrix multiply-accumulate operations.
- How it is used
- Accelerates FP8 matrix multiplication inside a precision-managed accumulation strategy.
- How deep it goes
- Explains Tensor Core execution, limited-precision accumulation, and periodic FP32 promotion, but not instruction syntax or programming code.
- Examples
- Uses y = Wx, FP8 activation and weight blocks, and diagrams of periodic partial-sum promotion.
- What you can do after
- Evaluate WGMMA's role in fast low-precision GEMM and explain why periodic FP32 accumulation improves numerical stability.
Read how Build a DeepSeek Model (From Scratch) explains WGMMA →
What to know first
- bfloat16bfloat16 is a 16-bit floating-point format with one sign bit, eight exponent bits, and seven fraction bits, providing a wide numerical range with reduced precision.
- matrix-matrix multiplicationMatrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
Related concepts
- column-major orderuses · Column-major order is a multidimensional array layout in which the leftmost dimension varies fastest, making each matrix column contiguous in memory.
- hardware registeruses · A hardware register is a small, rapidly accessible storage location within a processor or processing device that holds data, addresses, or control information during execution.
- CUTLASSused by · CUTLASS is NVIDIA’s collection of C++ templates for implementing high-performance linear algebra and tensor computations on GPUs.
Questions about WGMMA
What is WGMMA?
WGMMA is an NVIDIA GPU instruction family that performs matrix multiply-accumulate operations collectively across a warp group.
Which Manning books explain WGMMA?
CUDA for LLMs and Build a DeepSeek Model (From Scratch).
What should I know before learning WGMMA?
Manning books that teach WGMMA usually introduce bfloat16 and matrix-matrix multiplication first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

