liveBook · Manning

Synchronization

As explained in Grokking GPU Programming

Synchronization is the coordination of concurrent computations so that their execution observes required ordering or completion constraints.

Synchronization coordinates concurrent work so dependent operations happen in the right order. In CUDA, a kernel launch does not make the CPU wait; synchronize before the host relies on the kernel being finished. Within a block, __syncthreads() makes threads wait at a shared point. section 2.21 section 11.14

For shared-memory work, place barriers where threads depend on one another:

  • Wait after a thread writes data that another thread will read. section 11.4
  • Wait before overwriting data that other threads may still be reading. section 11.4
  • Keep every block thread on the same path to a barrier; do not put __syncthreads() inside a divergent branch. section 11.4
  • A block barrier does not synchronize different blocks. Use a kernel boundary when dependencies cross block boundaries. section 11.14

A reduction that sometimes gives the wrong sum

In a reduction over 1 through 256, threads combine values in shared memory over several steps. Without a barrier between steps, some threads may read before their partners have written; the result can vary between runs. Synchronizing after each step makes the sum reliably 32,896. section 5.22

tileA[ty][tx] = A[...]; tileB[ty][tx] = B[...];
__syncthreads();    #A
for (k...) sum += ...;   #B
__syncthreads();    #C

This tiled matrix-multiplication pattern shows barriers both after loading a tile and after using it, preventing premature reads and overwrites. section 12.11

helloKernel<<<1, 8>>>();
cudaDeviceSynchronize();
return 0;

This launch waits for the kernel before the program returns, avoiding an early exit while GPU work is still running. section 2.21

How Grokking GPU Programming covers it

How deep it goes
Explains host waits, block barriers, hazard-based placement, and kernel-boundary limits across blocks.
Examples
Uses kernel completion, a 1-to-256 reduction, and tiled matrix multiplication to show synchronization failures and fixes.
What you can do after
Diagnose missing synchronization by recognizing premature reads, overwrites, inconsistent results, and host exits before kernel completion.
How it connects
Connects CUDA kernels and shared memory with parallel reduction, scan, and tiled matrix multiplication.

Also explained in

Compare how 13 books explain synchronization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.