liveBook · Manning

Barrier

As explained in Grokking GPU Programming

A barrier is a synchronization mechanism that makes participating threads wait until the required threads reach a specified point.

When threads in a block share work, one may need to finish writing before others read or start the next step. A barrier makes every thread in that block wait until all have arrived. In CUDA, __syncthreads() also makes the block’s earlier shared and global memory writes visible to its threads after the barrier. section 5.14 section 14.10

Use the barrier where threads depend on one another’s work:

  • In a scan, place it after each round so no thread starts the next stride before its partners finish writing. section 6.18
  • In tiled matrix multiplication, use it after loading a tile and before threads compute with it. section 12.6
  • Every thread in the block must reach the same barrier. Putting it in a branch that some threads skip can hang the kernel or produce incorrect results. section 8.11
for (int stride = 1; stride < n; stride *= 2) {
    pout = 1 - pout;  pin = 1 - pout;
    /* add or copy into buf[pout][tid] */
    __syncthreads();    #A
}

Compare the scan loop with and without a barrier: the barrier at each round’s end keeps threads from reading unfinished results. section 6.18

A block barrier cannot synchronize different blocks. Ordinary kernel launches provide no in-kernel wait for all blocks; the kernel boundary is the point where all blocks are guaranteed to have finished. section 5.14

How Grokking GPU Programming covers it

How deep it goes
Explains block scope, memory visibility, placement, and common failure modes, but does not detail barrier implementation.
Examples
Uses scan rounds, matrix-tile loading, and conditional-barrier mistakes to illustrate correct placement and failure symptoms.
What you can do after
Diagnose missing or conditional barriers by recognizing intermittent results, stale cross-warp reads, or kernels that hang.
How it connects
Connects barriers to parallel scans, reductions, shared memory, and atomic operations.

Also explained in

Compare how 4 books explain barrier →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.