liveBook · Manning

Split-k matrix multiplication

As explained in GPU Programming with Triton

Split-K matrix multiplication partitions the contracting matrix dimension across program instances that produce partial results for shared output tiles.

Split-K matrix multiplication divides the contracting dimension among program instances that compute partial results for shared output tiles. Atomic additions combine those partials at shared output addresses. section 4.1.3

For accumulation where only the final sum matters, relaxed atomic ordering preserves atomicity without ordering surrounding memory accesses. Floating-point addition is not associative, so changing the order of partial additions can change the result’s bits. section 4.1.3

tl.atomic_add(pointer, val, mask=None, sem=None, scope=None)

Atomic addition is the operation used to combine partial results at a shared output address. section 4.1.3

How GPU Programming with Triton covers it

How deep it goes
Explains relaxed versus acquire-release ordering and run-to-run bitwise variation, but gives no complete Split-K kernel.
Examples
Uses four program instances adding partial results to one output address, with completion order changing across runs.
What you can do after
Apply relaxed atomic ordering when only the final sum matters, avoiding ordering constraints on surrounding memory accesses.
How it connects
Links matrix multiplication to atomic operations, memory ordering, and floating-point addition.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.