liveBook · Manning

Atomicity

As explained in GPU Programming with Triton

An atomic operation performs a memory read-modify-write update as one indivisible transaction that cannot be interleaved at the target address.

An atomic operation performs a memory read-modify-write as one indivisible transaction. Competing updates to the same address cannot interleave and overwrite one another’s results. This matters when GPU program instances contribute to a shared output without a fixed execution order. section 4.1.2 section 4.3

Figure 4.4 A non-atomic read-modify-write allows another thread to intervene between the read and write steps, corrupting the result. An atomic operation encloses the entire sequence in a single hardware transaction that blocks other threads from accessing the same address until the operation completes
CH04 F04 Fartale

Compare an interleaved read-modify-write with an atomic transaction. section 4.1.2

In Triton, a reduction such as tl.sum combines values within one program instance, not across the launch grid. To compute a global sum, each instance can reduce its tile locally and atomically add the partial result to a shared output. Alternatively, a first kernel can write partial results and a second kernel can reduce them. section 4.1.2 section 4.3

Listing 4.3 One-pass global sum using atomic add
import torch
import triton
import triton.language as tl

@triton.jit
def atomic_sum_kernel(x_ptr, out_ptr, n_elements,
                      BLOCK_SIZE: tl.constexpr):
    pid = tl.program_id(0)
    offsets = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE)
    mask = offsets < n_elements

    x = tl.load(x_ptr + offsets, mask=mask, other=0.0)
    part = tl.sum(x, axis=0, dtype=tl.float32) #1
    tl.atomic_add(out_ptr, part) #2


def sum_atomic(x: torch.Tensor, block_size: int = 1024):
    assert x.is_cuda and x.ndim == 1
    n = x.numel()

    out = torch.zeros((1,), device=x.device, dtype=torch.float32) #3
    grid = (triton.cdiv(n, block_size),)
    atomic_sum_kernel[grid](x, out, n, BLOCK_SIZE=block_size)
    return out[0]

The one-pass sum masks out-of-range loads, reduces each tile, and uses tl.atomic_add to combine partials into an output initialized to zero. section 4.1.2

A global-memory atomic is resolved at the L2 cache. Updates to the same address serialize, so many instances targeting one output form a queue. Reducing a tile first limits the kernel to one atomic update per instance rather than one per element. Updates to different addresses can proceed in parallel, making contention an important part of the design tradeoff. section 4.1.3 section 4.3

Atomicity prevents lost updates, but it does not fix their order. Floating-point partial sums can arrive in different orders across runs; because floating-point addition is not associative, results may differ bitwise for identical inputs. section 4.1.3 section 4.3

Figure 4.5 Four program instances atomically add their partial results to the same output address. The hardware processes them in whatever order the SMs happen to finish, and that order can change between runs. Because floating-point addition is not associative, the two runs produce different bitwise results despite identical inputs
CH04 F05 Fartale

See how different completion orders change a floating-point atomic sum. section 4.1.3

Atomic updates are useful when multiple instances can contribute to the same data-dependent destination:

  • Split-K matrix multiplication combines partial results. section 4.3
  • Scatter operations, including an embedding backward pass, update shared destinations. section 4.3
  • Histogram binning accumulates counts into bins. section 4.3

How GPU Programming with Triton covers it

How deep it goes
Explains read-modify-write execution at L2, contention, local aggregation, and floating-point ordering effects.
Examples
Demonstrates a one-pass global sum and contrasts it with a two-pass reduction.
What you can do after
Apply it via Triton's tl.atomic_add to combine per-instance partial sums.
How it connects
Connects atomicity to reductions, scatter operations, histogram binning, and Split-K matrix multiplication.

Also explained in

Compare how 19 books explain atomicity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.