liveBook · Manning

Tl.atomic_add

As explained in GPU Programming with Triton

tl.atomic_add is a Triton API operation that atomically adds a value to a global-memory location and returns its previous value.

`tl.atomic_add` adds a value to a memory location as one indivisible read-modify-write operation and returns the previous value. It lets Triton program instances safely accumulate into a shared global-memory address without losing updates. section 4.1.2 section 4.1.3

Without an atomic operation, two program instances could read the same old value, add their contributions, and overwrite each other’s results. An atomic add coordinates the whole update at the address. Updates to the same address serialize; requests to different addresses can proceed in parallel. section 4.1.2 section 4.3

Listing 4.3 One-pass global sum using atomic add
import torch
import triton
import triton.language as tl

@triton.jit
def atomic_sum_kernel(x_ptr, out_ptr, n_elements,
                      BLOCK_SIZE: tl.constexpr):
    pid = tl.program_id(0)
    offsets = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE)
    mask = offsets < n_elements

    x = tl.load(x_ptr + offsets, mask=mask, other=0.0)
    part = tl.sum(x, axis=0, dtype=tl.float32) #1
    tl.atomic_add(out_ptr, part) #2


def sum_atomic(x: torch.Tensor, block_size: int = 1024):
    assert x.is_cuda and x.ndim == 1
    n = x.numel()

    out = torch.zeros((1,), device=x.device, dtype=torch.float32) #3
    grid = (triton.cdiv(n, block_size),)
    atomic_sum_kernel[grid](x, out, n, BLOCK_SIZE=block_size)
    return out[0]

A one-pass global sum reduces each tile locally, then atomically adds its partial sum to a zero-initialized output. section 4.1.2

The global-sum kernel masks loads beyond the input length and substitutes zero, then uses `tl.sum` to produce one partial per program instance. The call `tl.atomic_add(out_ptr, part)` combines those partials. Local reduction matters: 256 instances each processing 1,024 values issue 256 atomics instead of 262,144, greatly reducing contention at the output address. section 4.1.2 section 4.1.3

A two-pass reduction is an alternative: write each partial to its own global-memory location, then launch another kernel to reduce the partials. Atomic adds are particularly useful when the output address depends on data rather than a fixed grid position, as in scatter operations and histogram binning. section 4.1.2 section 4.3

Atomic correctness does not guarantee bitwise reproducibility for floating-point sums. Program instances can finish in different orders, and floating-point addition is not associative. Contention also has a cost: requests to one address queue up. Native atomic support depends on the data type and GPU generation; unsupported types may use a slower compare-and-swap loop. section 4.1.3 section 4.3

Figure 4.5 Four program instances atomically add their partial results to the same output address. The hardware processes them in whatever order the SMs happen to finish, and that order can change between runs. Because floating-point addition is not associative, the two runs produce different bitwise results despite identical inputs
CH04 F05 Fartale

The accumulation order can change between runs, producing bitwise-different floating-point results. section 4.1.3

How GPU Programming with Triton covers it

How deep it goes
Explains indivisible updates, contention, local reduction, floating-point ordering, and hardware support, with a complete one-pass sum kernel.
Examples
Demonstrates a masked one-dimensional global sum with one atomic update per program instance and a zero-initialized output.
What you can do after
Apply it via Triton to accumulate program-instance partials safely while accounting for contention and floating-point ordering.
How it connects
Connects atomic addition to global reduction, two-pass reduction, scatter operations, histogram binning, and Split-K matrix multiplication.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.