A CUDA kernel launch is asynchronous: the CPU starts GPU work without waiting for it to finish. The GPU can process data while the CPU continues; cudaDeviceSynchronize() makes the CPU wait until the GPU work is done. section 2.17
Squaring array elements
A kernel launches one thread per element to square values in device memory. Calling cudaDeviceSynchronize() after the launch gives the CPU a clear wait point, so kernel errors surface there during debugging. A blocking device-to-host copy also waits for earlier GPU work, so an extra synchronization before that copy is not required. section 2.17
Figure 2.9 Step 3 — Compute: the GPU takes over. The CPU launches the kernel and steps back to wait; all four threads fire at once, each one reading its value from device memory, squaring it, and writing the result back (the up and down arrows), all in parallel. When the last thread finishes, cudaDeviceSynchronize() returns and the CPU carries on.

Trace the CPU launching the kernel, GPU threads processing values, and the CPU resuming after synchronization. section 2.17




































