liveBook · Manning

__syncthreads()

As explained in Grokking GPU Programming

__syncthreads is a CUDA barrier operation that makes threads in one block wait until all threads in that block reach the barrier.

Suppose threads are filling a shared array, and each thread needs a value written by a neighbor. A fast thread could start reading before its neighbor has finished writing. __syncthreads() prevents this: it is a CUDA barrier that makes every thread in a block wait until all threads in that block reach it. section 5.14 section 11.4

Think of a relay race in which nobody starts the next lap until every runner finishes the current one. The block waits for its slowest thread, but that wait protects correctness. The barrier also makes shared and global memory writes made by the block’s threads before it visible to those threads afterward. section 5.14 section 14.10

The scope is one block, not the entire grid. Blocks can run in an unpredictable order, so __syncthreads() cannot coordinate them. A kernel boundary guarantees that all blocks from the launch have finished; cross-block dependencies can also require a different tool, such as atomicAdd for combining reduction results. section 5.14 section 11.14

Place a barrier between dependent phases: after shared-memory writes that other threads will read, or before overwriting slots that other threads may still be reading. A loop alone does not require a barrier; the cross-thread dependency does. section 11.4

for (int stride = blockDim.x / 2; stride > 0; stride >>= 1) {
    if (tid < stride) {
        sdata[tid] += sdata[tid + stride];
    }
    __syncthreads();    #A
}

In this reduction, only threads with tid < stride update values, but every thread reaches the barrier before the next reduction step reads them. section 5.22

A missing barrier

Summing the integers 1 through 256 should give 32,896. Without a barrier after each reduction step, a thread may read a value its partner has not written yet. The kernel can compile and run while producing a smaller or inconsistent sum. Synchronizing after each step fixes that dependency. section 5.22

Every thread in the block must reach the same barrier, even if it has no useful work in that phase. A barrier inside a branch entered by only some threads causes undefined behavior: the kernel may hang or return wrong results. Keep the conditional work inside the branch and put the barrier on the common path. section 6.10 section 8.11

if (tid < 128) {
    // some work...
}
__syncthreads();    #B

Here the work is conditional, but the barrier is outside the branch, so all block threads can reach it. section 8.11

How Grokking GPU Programming covers it

How deep it goes
Explains barrier participation, memory visibility, block scope, placement hazards, and common failure symptoms.
Examples
Uses a 256-element sum, a shared-memory scan load, and a conditional histogram barrier to illustrate synchronization.
What you can do after
Diagnose missing or partially reached barriers and place synchronization between dependent block-level phases.
How it connects
Connects barriers to shared memory, reduction, scan, memory fences, and cross-warp races.

Also explained in

Compare how 2 books explain __syncthreads() →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.