liveBook · Manning

Warp divergence

As explained in Grokking GPU Programming

Branch divergence is a GPU execution inefficiency that occurs when threads in the same warp take different control-flow paths.

A GPU kernel can slow down when neighboring threads need to do different jobs. On an NVIDIA GPU, threads run in groups of 32 called warps. The hardware issues instructions to a whole warp at once, so a warp works most efficiently when its threads follow the same path. section 14.0 section 14.1

Warp divergence, also called branch divergence, occurs when threads within a warp take different control-flow paths. The GPU runs those paths one at a time, masking threads that do not belong to the current path. Those threads wait until their path runs. section 14.2

An even–odd split

Suppose threads 0–31 branch on whether their index is even. First, the 16 even-indexed threads run path A while the odd-indexed threads are masked. Then the 16 odd-indexed threads run path B while the even-indexed threads wait. The warp pays for both paths. section 14.2

Figure 14.2 A divergent branch inside a single warp, with time flowing left to right. When the threads take different paths, the warp serializes them: first the active (solid) threads run expensive_A while the masked (dashed) threads wait, then the roles flip and the remaining threads run expensive_B. Both paths run one after the other, so the warp pays for both.

Follow the two paths over time to see which threads are active and which are masked. section 14.2

Only differences within a warp cause this divergence penalty. If every thread in one warp takes path A and every thread in another takes path B, each warp remains unified. Where possible, group work by path so threads in the same warp agree, without changing the required results. section 14.2

In a simple timing model, the costs of paths taken by a warp add up. Two equally expensive paths can therefore take about twice as long as one; four can take about four times as long. These comparisons describe the model, not a universal kernel speedup. section 14.2

How Grokking GPU Programming covers it

How deep it goes
Explains masking and serialized branch paths, with a simple additive timing model.
Examples
Uses an even–odd branch and a four-way branch to illustrate active and masked threads.
What it assumes
Assumes familiarity with CUDA kernels and threads while explaining warp execution in plain language.
What you can do after
Diagnose/debug potential divergence by checking whether threads within the same warp take different paths.
How it connects
Connects branch divergence to warps and the single-instruction, multiple-thread execution model.

Also explained in

Compare how 2 books explain warp divergence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.