liveBook · Manning

Matrix-matrix multiplication

As explained in GPU Programming with Triton

Matrix multiplication combines two compatible matrices by taking row-column dot products.

For A with shape M × K and B with shape K × N, the product C = AB has shape M × N. Each C[i,j] is the dot product of row i of A and column j of B; the shared K dimension must match. section 5.5

C[i, j] = sum over p in [0, K) of A[i, p] * B[p, j]

Each output entry sums pairwise products across the shared K dimension. section 5.5

A scalar-output kernel computes each entry independently, repeating reads of input values. A tiled kernel computes several outputs together, reuses loaded tiles of A and B, and uses tl.dot to contract their K dimensions. Masks handle incomplete edge tiles; grouped program ordering and autotuning can improve performance. section 5.10 section 5.6.1

A small product

For A = [[1, 2, 3], [4, 5, 6]] and B = [[7, 8], [9, 10], [11, 12]], the result is [[58, 64], [139, 154]]. For example, the first row’s second output is 1 × 8 + 2 × 10 + 3 × 12 = 64. section 5.5

Matrix products recur in neural-network linear layers, attention projections, and training gradients. GEMM generalizes the operation to C ← αAB + βC, scaling both the new product and any existing output values. section 5.5 section 5.5.1

How GPU Programming with Triton covers it

How deep it goes
Derives the scalar contraction, then develops tiled tl.dot kernels with masking, data reuse, and autotuning.
Examples
Uses a 2×3-by-3×2 numerical product, attention projections, and accumulated weight gradients.
What it assumes
Extends one-dimensional layouts by teaching matrix addressing, strides, tile shapes, and program ordering.
What you can do after
Build it from scratch by mapping output tiles to programs, reusing input blocks, and contracting them with tl.dot.
How it connects
Links matrix multiplication to dot products, GEMM, Tensor Cores, Triton, and neural networks.

Also explained in

Compare how 13 books explain matrix-matrix multiplication →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.