A CUDA kernel launch returns immediately, so the host can continue while the device is still working. Call cudaDeviceSynchronize() when the host must wait for pending device work to finish—before relying on results or letting the program exit. section 2.5
Use synchronization where completion matters:
- Wait before the host reads results or exits; otherwise output may be missing or incomplete. section 2.21
- A synchronization point can also report an error from work that failed during an asynchronous kernel; a bad launch configuration is instead checked with cudaGetLastError() after launch. section 2.5
- Use synchronization sparingly when tuning performance. section 3.16
Wait before exiting
After launching helloKernel<<<1, 8>>>(), the host could reach return before the eight threads finish printing. Adding cudaDeviceSynchronize() before return makes the host wait. Output may appear complete without it on swforces.com, but real-GPU execution is asynchronous. section 2.1 section 2.21
Figure 2.1 Launch, then wait. The CPU fires the kernel and keeps going while the threads fan out and run in parallel. Threads finish at different times; cudaDeviceSynchronize() makes the CPU wait until the last thread finishes.

The launch-then-wait diagram shows the host waiting for the last thread to finish. section 2.5

