liveBook · Manning

Matrix multiply-accumulate

As explained in GPU Programming with Triton

A matrix multiply-accumulate operation multiplies two matrices and adds a third matrix to the product.

A matrix multiply-accumulate (MMA) computes D = AB + C. For A shaped [M, K] and B shaped [K, N], C and D are shaped [M, N]. section 5.7

\[D_{i,j} = C_{i,j} + \sum_{k=0}^{K-1} A_{i,k}B_{k,j}.\]

Each output element adds the dot product of one row of A and one column of B to the corresponding accumulator element. section 5.7

Dense matrix multiplication repeats this pattern across many outputs. NVIDIA Tensor Cores execute matrix multiply-accumulates in hardware, while Triton’s tl.dot expresses the operation on blocks. Each call contracts matching K dimensions; repeated calls can accumulate successive K slices. Loads, address calculations, boundary handling, and stores remain explicit in the surrounding kernel. section 5.7 section 5.6.1

How GPU Programming with Triton covers it

How deep it goes
Defines the operation, shapes, and tiled accumulation, while leaving layout and tile requirements target-dependent.
Examples
Compares scalar FMA with a [64, 32] by [32, 64] workload and shows accumulation across K slices.
What it assumes
Builds from scalar fused multiply-add and explains matrix dimensions without assuming prior matrix-instruction knowledge.
What you can do after
Apply it via Triton's tl.dot to accumulate tiled products across K slices and produce an output tile.
How it connects
Connects Tensor Cores and Triton's tl.dot to scalar fused multiply-add and CUDA-core arithmetic.

Also explained in

Compare how 2 books explain matrix multiply-accumulate →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.