Synchronization coordinates concurrent work so dependent operations happen in the right order. In CUDA, a kernel launch does not make the CPU wait; synchronize before the host relies on the kernel being finished. Within a block, __syncthreads() makes threads wait at a shared point. section 2.21 section 11.14
For shared-memory work, place barriers where threads depend on one another:
- Wait after a thread writes data that another thread will read. section 11.4
- Wait before overwriting data that other threads may still be reading. section 11.4
- Keep every block thread on the same path to a barrier; do not put __syncthreads() inside a divergent branch. section 11.4
- A block barrier does not synchronize different blocks. Use a kernel boundary when dependencies cross block boundaries. section 11.14
A reduction that sometimes gives the wrong sum
In a reduction over 1 through 256, threads combine values in shared memory over several steps. Without a barrier between steps, some threads may read before their partners have written; the result can vary between runs. Synchronizing after each step makes the sum reliably 32,896. section 5.22
tileA[ty][tx] = A[...]; tileB[ty][tx] = B[...]; __syncthreads(); #A for (k...) sum += ...; #B __syncthreads(); #C
This tiled matrix-multiplication pattern shows barriers both after loading a tile and after using it, preventing premature reads and overwrites. section 12.11
helloKernel<<<1, 8>>>(); cudaDeviceSynchronize(); return 0;
This launch waits for the kernel before the program returns, avoiding an early exit while GPU work is still running. section 2.21












