When many GPU program instances atomically update the same address, the updates serialize; updates to different addresses can proceed in parallel. A single global-memory atomic costs roughly an L2 cache access—about 200 cycles on current hardware—but low contention can hide that latency. Under high contention, 1,000 atomics to one address can form a chain of roughly 200,000 cycles. section 4.1.3
Reduce contention by combining contributions before writing to shared output:
- Use a block-level `tl.sum` to collapse a tile into one scalar, then issue one `tl.atomic_add` per program instance. For 256 instances processing 1,024 values each, this replaces 262,144 atomics with 256. section 4.1.3
tl.atomic_add(pointer, val, mask=None, sem=None, scope=None)
Triton’s `tl.atomic_add` interface for applying the update. section 4.1.3
