liveBook · Manning

Torch.cuda.synchronize()

As explained in GPU Programming with Triton

torch.cuda.synchronize() is a PyTorch CUDA API call that blocks the CPU until all previously issued GPU work has completed.

GPU kernel launches return before the GPU necessarily finishes its work. Call torch.cuda.synchronize() before starting and after stopping a CPU wall-clock timer: the first ensures the GPU is idle at the start, and the second waits for queued work to complete. section 2.2.10

import time
import torch

torch.cuda.synchronize()
start = time.time()
z = add(x, y)
torch.cuda.synchronize()
end = time.time()
print(f"elapsed: {end - start:.6f} s")

This timing pattern brackets a Triton kernel call with synchronization so the measured interval includes GPU execution. section 2.2.10

Without synchronization, a wall-clock measurement may capture mostly CPU launch time. CUDA events offer another approach, measuring elapsed time on the GPU timeline. section 2.2.10

How GPU Programming with Triton covers it

How deep it goes
Explains CPU/GPU timing boundaries and pre- and post-kernel synchronization, then contrasts CUDA events.
Examples
Compares unsynchronized and synchronized wall-clock timings for Triton vector addition.
What it assumes
Assumes a launched GPU kernel and CPU wall-clock timing, and clarifies when the GPU has completed queued work.
What you can do after
Apply it via PyTorch to measure GPU work completed within a wall-clock interval.
How it connects
Connects GPU kernel benchmarking with asynchronous execution and CUDA events.

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.