liveBook · Manning

Memory ordering

As explained in GPU Programming with Triton

Memory ordering semantics specify which memory operations may be reordered around an atomic operation and when their effects become visible.

Memory ordering controls which other memory operations may move across an atomic operation and when their effects are visible. Triton atomics accept "relaxed", "acquire", "release", and "acq_rel"; "acq_rel" is the default and acts as a fence, preventing memory operations from the current thread from being reordered across the atomic. section 4.1.3

Choose ordering according to whether surrounding memory accesses need constraints:

  • "relaxed" guarantees the atomic read-modify-write itself but places no ordering constraints on other memory accesses. It is sufficient for simple accumulation when only the final sum matters, and avoids fence overhead. section 4.1.3
  • The atomic scope is separate from ordering: "gpu" covers the device, "cta" the current thread block, and "sys" the system, including CPU and peer GPUs. section 4.1.3
tl.atomic_add(pointer, val, mask=None, sem=None, scope=None)

The atomic_add signature exposes sem and scope as arguments for choosing ordering and visibility. section 4.1.3

How GPU Programming with Triton covers it

How deep it goes
Defines four ordering choices and explains relaxed and default acquire-release behavior, without detailing each option individually.
Examples
Uses simple accumulation and Split-K GEMM to illustrate when relaxed ordering is sufficient.
What you can do after
Configure Triton atomic ordering for accumulation patterns that do not require surrounding memory accesses to be ordered.
How it connects
Connects atomic operations with memory fences and Split-K GEMM.

Also explained in

Compare how 3 books explain memory ordering →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.