liveBook · Manning

Memory contention

As explained in GPU Programming with Triton

Competition among concurrent operations for access to the same memory location or cache line.

When many GPU program instances atomically update the same address, the updates serialize; updates to different addresses can proceed in parallel. A single global-memory atomic costs roughly an L2 cache access—about 200 cycles on current hardware—but low contention can hide that latency. Under high contention, 1,000 atomics to one address can form a chain of roughly 200,000 cycles. section 4.1.3

Reduce contention by combining contributions before writing to shared output:

  • Use a block-level `tl.sum` to collapse a tile into one scalar, then issue one `tl.atomic_add` per program instance. For 256 instances processing 1,024 values each, this replaces 262,144 atomics with 256. section 4.1.3
tl.atomic_add(pointer, val, mask=None, sem=None, scope=None)

Triton’s `tl.atomic_add` interface for applying the update. section 4.1.3

How GPU Programming with Triton covers it

How deep it goes
Quantifies serialization costs and works through tile reduction before atomic updates.
Examples
Compares 1,000 updates to one address with updates to different addresses, then reduces 262,144 atomics to 256.
What you can do after
Apply Triton block-level reduction before `tl.atomic_add` to reduce competing updates to a shared address.
How it connects
Connects global-memory atomics with L2 cache behavior and block-level reduction.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.