An atomic operation performs a memory read-modify-write as one indivisible transaction. Competing updates to the same address cannot interleave and overwrite one another’s results. This matters when GPU program instances contribute to a shared output without a fixed execution order. section 4.1.2 section 4.3
Figure 4.4 A non-atomic read-modify-write allows another thread to intervene between the read and write steps, corrupting the result. An atomic operation encloses the entire sequence in a single hardware transaction that blocks other threads from accessing the same address until the operation completes
Compare an interleaved read-modify-write with an atomic transaction. section 4.1.2
In Triton, a reduction such as tl.sum combines values within one program instance, not across the launch grid. To compute a global sum, each instance can reduce its tile locally and atomically add the partial result to a shared output. Alternatively, a first kernel can write partial results and a second kernel can reduce them. section 4.1.2 section 4.3
Listing 4.3 One-pass global sum using atomic add
import torch
import triton
import triton.language as tl
@triton.jit
def atomic_sum_kernel(x_ptr, out_ptr, n_elements,
BLOCK_SIZE: tl.constexpr):
pid = tl.program_id(0)
offsets = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE)
mask = offsets < n_elements
x = tl.load(x_ptr + offsets, mask=mask, other=0.0)
part = tl.sum(x, axis=0, dtype=tl.float32) #1
tl.atomic_add(out_ptr, part) #2
def sum_atomic(x: torch.Tensor, block_size: int = 1024):
assert x.is_cuda and x.ndim == 1
n = x.numel()
out = torch.zeros((1,), device=x.device, dtype=torch.float32) #3
grid = (triton.cdiv(n, block_size),)
atomic_sum_kernel[grid](x, out, n, BLOCK_SIZE=block_size)
return out[0]
The one-pass sum masks out-of-range loads, reduces each tile, and uses tl.atomic_add to combine partials into an output initialized to zero. section 4.1.2
A global-memory atomic is resolved at the L2 cache. Updates to the same address serialize, so many instances targeting one output form a queue. Reducing a tile first limits the kernel to one atomic update per instance rather than one per element. Updates to different addresses can proceed in parallel, making contention an important part of the design tradeoff. section 4.1.3 section 4.3
Atomicity prevents lost updates, but it does not fix their order. Floating-point partial sums can arrive in different orders across runs; because floating-point addition is not associative, results may differ bitwise for identical inputs. section 4.1.3 section 4.3
Figure 4.5 Four program instances atomically add their partial results to the same output address. The hardware processes them in whatever order the SMs happen to finish, and that order can change between runs. Because floating-point addition is not associative, the two runs produce different bitwise results despite identical inputs
See how different completion orders change a floating-point atomic sum. section 4.1.3
Atomic updates are useful when multiple instances can contribute to the same data-dependent destination:
- Split-K matrix multiplication combines partial results. section 4.3
- Scatter operations, including an embedding backward pass, update shared destinations. section 4.3
- Histogram binning accumulates counts into bins. section 4.3


















