liveBook · Manning

Scatter operation

As explained in GPU Programming with Triton

A scatter operation updates output locations selected by per-element indices or data-dependent addresses.

Scatter operations update output locations selected by input indices. When multiple inputs target the same location, atomic updates preserve correctness, but heavy collisions serialize at the L2 cache. Reduce values within each program instance before issuing atomic writes to limit contention. section 4.1.3

Common scatter workloads include:

  • PyTorch scatter_add and index_add, and nn.Embedding’s backward pass, where repeated indices require summing updates to the same output. section 4.1.3
  • Histogram and binning kernels, where input values determine output addresses that may be shared. section 4.1.3
tl.atomic_add(pointer, val, mask=None, sem=None, scope=None)

Triton’s atomic-add interface accepts a pointer, value, optional mask, memory semantics, and scope. section 4.1.3

How GPU Programming with Triton covers it

How deep it goes
Explains collision costs and recommends local reduction before atomic writes, without walking through a complete scatter kernel.
Examples
Uses scatter_add, index_add, embedding backward, histogram computation, and binning as examples.
What you can do after
Apply local reduction before issuing atomic writes in custom Triton scatter kernels.
How it connects
Connects scatter operations to atomic operations, reductions, histograms, binning, and embedding gradients.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.