`tl.atomic_add` adds a value to a memory location as one indivisible read-modify-write operation and returns the previous value. It lets Triton program instances safely accumulate into a shared global-memory address without losing updates. section 4.1.2 section 4.1.3
Without an atomic operation, two program instances could read the same old value, add their contributions, and overwrite each other’s results. An atomic add coordinates the whole update at the address. Updates to the same address serialize; requests to different addresses can proceed in parallel. section 4.1.2 section 4.3
Listing 4.3 One-pass global sum using atomic add
import torch
import triton
import triton.language as tl
@triton.jit
def atomic_sum_kernel(x_ptr, out_ptr, n_elements,
BLOCK_SIZE: tl.constexpr):
pid = tl.program_id(0)
offsets = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE)
mask = offsets < n_elements
x = tl.load(x_ptr + offsets, mask=mask, other=0.0)
part = tl.sum(x, axis=0, dtype=tl.float32) #1
tl.atomic_add(out_ptr, part) #2
def sum_atomic(x: torch.Tensor, block_size: int = 1024):
assert x.is_cuda and x.ndim == 1
n = x.numel()
out = torch.zeros((1,), device=x.device, dtype=torch.float32) #3
grid = (triton.cdiv(n, block_size),)
atomic_sum_kernel[grid](x, out, n, BLOCK_SIZE=block_size)
return out[0]
A one-pass global sum reduces each tile locally, then atomically adds its partial sum to a zero-initialized output. section 4.1.2
The global-sum kernel masks loads beyond the input length and substitutes zero, then uses `tl.sum` to produce one partial per program instance. The call `tl.atomic_add(out_ptr, part)` combines those partials. Local reduction matters: 256 instances each processing 1,024 values issue 256 atomics instead of 262,144, greatly reducing contention at the output address. section 4.1.2 section 4.1.3
A two-pass reduction is an alternative: write each partial to its own global-memory location, then launch another kernel to reduce the partials. Atomic adds are particularly useful when the output address depends on data rather than a fixed grid position, as in scatter operations and histogram binning. section 4.1.2 section 4.3
Atomic correctness does not guarantee bitwise reproducibility for floating-point sums. Program instances can finish in different orders, and floating-point addition is not associative. Contention also has a cost: requests to one address queue up. Native atomic support depends on the data type and GPU generation; unsupported types may use a slower compare-and-swap loop. section 4.1.3 section 4.3
Figure 4.5 Four program instances atomically add their partial results to the same output address. The hardware processes them in whatever order the SMs happen to finish, and that order can change between runs. Because floating-point addition is not associative, the two runs produce different bitwise results despite identical inputs
The accumulation order can change between runs, producing bitwise-different floating-point results. section 4.1.3
