liveBook · Manning

Synchronization

As explained in GPU Programming with Triton

Synchronization coordinates concurrent execution so dependent operations observe completed memory writes in the required order.

A reduction needs synchronization when one thread consumes a partial result written by another. It ensures producers finish writing before consumers read, preventing incomplete values. section 4.0

The required coordination depends on where the cooperating threads run:

  • Within a warp, lanes exchange values through register shuffles. section 4.1
  • Across warps in a block, threads use shared memory and explicit barriers. section 4.1
  • Across blocks, threads cannot synchronize within one kernel launch; coordination uses global memory, usually with another launch or atomics. section 4.1
Figure 4.2 The communication hierarchy for reductions on a GPU. Lanes within a warp exchange values through fast register shuffles. Warps within a block communicate through shared memory with explicit barriers. Blocks cannot synchronize inside a single kernel and must coordinate through global memory
CH04 F02 Fartale

Shows how reduction communication and synchronization differ across GPU execution scopes. section 4.1

How GPU Programming with Triton covers it

How deep it goes
Maps synchronization needs across warp, block, and grid scopes, but does not detail Triton synchronization constructs.
Examples
Uses reductions for gradient sums, softmax denominators, and normalization statistics as motivating workloads.
What you can do after
Understand when reductions require synchronization and which GPU communication scope supports their cooperating threads.
How it connects
Connects reductions with register shuffles, shared memory, global memory, kernel launches, and atomic operations.

Also explained in

Compare how 13 books explain synchronization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.