liveBook · Manning

Warp divergence

As explained in GPU Programming with Triton

Warp divergence is the execution of different conditional branches by threads in the same warp, requiring the branches to be processed sequentially.

Warp divergence occurs when threads in the same warp take different branches of a conditional. In NVIDIA’s 32-thread warp model, threads execute in lockstep, so the hardware processes each branch in turn and disables lanes that do not match it. section 3.4 section 3.4.1

A divergent ReLU branch

For `if x < 0: output = 0; else: output = x`, negative and positive inputs can send lanes down different paths. The warp executes both branches serially; if each takes 10 cycles, the divergent warp takes 20 cycles rather than 10. A split between branches can roughly halve throughput, while lanes taking the same branch avoid this penalty. section 3.4.1

Figure 3.6 When threads in the same warp evaluate an if/else condition differently, the GPU must execute both branches serially. During each branch, only a subset of lanes is active while the others are idle, reducing effective parallelism and throughput.
CH03 F06 Fartale

The figure shows how serial branch execution leaves some lanes inactive during each path. section 3.4.1

For element-wise conditions in Triton, `tl.where` selects a value per lane while keeping the warp on one control-flow path. Both candidate values are evaluated, however, so expensive candidate computations can still cost time. An `if` remains suitable when its condition is shared by every lane in the block. section 3.4.2

output = tl.where(x > 0, x, tl.zeros_like(x))

This ReLU expression uses `tl.where` to select between the input and zero for each lane. section 3.4.2

How GPU Programming with Triton covers it

How deep it goes
Explains branch serialization, inactive lanes, throughput loss, and tl.where’s tradeoffs.
Examples
Uses ReLU’s negative-versus-positive branches and a 10-cycle-per-branch illustration.
What it assumes
Builds from familiar if-else conditions and introduces NVIDIA’s 32-thread warp behavior.
What you can do after
Apply Triton’s tl.where to select values per lane while accounting for candidate computations.
How it connects
Connects conditional branching with Triton’s tl.where, ReLU, and per-lane masks.

Also explained in

Compare how 2 books explain warp divergence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.