Suppose threads are filling a shared array, and each thread needs a value written by a neighbor. A fast thread could start reading before its neighbor has finished writing. __syncthreads() prevents this: it is a CUDA barrier that makes every thread in a block wait until all threads in that block reach it. section 5.14 section 11.4
Think of a relay race in which nobody starts the next lap until every runner finishes the current one. The block waits for its slowest thread, but that wait protects correctness. The barrier also makes shared and global memory writes made by the block’s threads before it visible to those threads afterward. section 5.14 section 14.10
The scope is one block, not the entire grid. Blocks can run in an unpredictable order, so __syncthreads() cannot coordinate them. A kernel boundary guarantees that all blocks from the launch have finished; cross-block dependencies can also require a different tool, such as atomicAdd for combining reduction results. section 5.14 section 11.14
Place a barrier between dependent phases: after shared-memory writes that other threads will read, or before overwriting slots that other threads may still be reading. A loop alone does not require a barrier; the cross-thread dependency does. section 11.4
for (int stride = blockDim.x / 2; stride > 0; stride >>= 1) {
if (tid < stride) {
sdata[tid] += sdata[tid + stride];
}
__syncthreads(); #A
}In this reduction, only threads with tid < stride update values, but every thread reaches the barrier before the next reduction step reads them. section 5.22
A missing barrier
Summing the integers 1 through 256 should give 32,896. Without a barrier after each reduction step, a thread may read a value its partner has not written yet. The kernel can compile and run while producing a smaller or inconsistent sum. Synchronizing after each step fixes that dependency. section 5.22
Every thread in the block must reach the same barrier, even if it has no useful work in that phase. A barrier inside a branch entered by only some threads causes undefined behavior: the kernel may hang or return wrong results. Keep the conditional work inside the branch and put the barrier on the common path. section 6.10 section 8.11
if (tid < 128) {
// some work...
}
__syncthreads(); #BHere the work is conditional, but the barrier is outside the branch, so all block threads can reach it. section 8.11

