Warp divergence
Warp divergence occurs when threads in a GPU warp follow different control-flow paths, requiring the paths to be executed sequentially.
Grokking GPU Programming
Branch divergence is a GPU execution inefficiency that occurs when threads in the same warp take different control-flow paths.
- How it is used
- Explains a hidden GPU kernel slowdown through the execution of threads in warps.
- How deep it goes
- Explains masking and serialized branch paths, with a simple additive timing model.
- Examples
- Uses an even–odd branch and a four-way branch to illustrate active and masked threads.
- What you can do after
- Diagnose/debug potential divergence by checking whether threads within the same warp take different paths.
Read how Grokking GPU Programming explains warp divergence →
GPU Programming with Triton
Warp divergence is the execution of different conditional branches by threads in the same warp, requiring the branches to be processed sequentially.
- How it is used
- Diagnoses a GPU performance cost caused by lane-varying branches and motivates Triton’s per-lane selection.
- How deep it goes
- Explains branch serialization, inactive lanes, throughput loss, and tl.where’s tradeoffs.
- Examples
- Uses ReLU’s negative-versus-positive branches and a 10-cycle-per-branch illustration.
- What you can do after
- Apply Triton’s tl.where to select values per lane while accounting for candidate computations.
Read how GPU Programming with Triton explains warp divergence →
What to know first
- parallelismParallelism is the execution of multiple independent tasks or computational operations at the same time rather than sequentially.
- CPUA central processing unit (CPU) is a computer processor that executes program instructions and performs general-purpose computational operations.
- memory latencyMemory latency is the time between requesting data from a memory system and receiving the requested data.
- warpA warp is a group of GPU threads scheduled and issued together for execution, typically processing the same instruction on different data.
Related concepts
- sorting algorithmaddressed by · A sorting algorithm is a procedure that rearranges elements of a collection into an order determined by an ordering criterion.
- warp-uniform branchaddressed by · A warp-uniform branch is a conditional branch for which all threads in a warp take the same path.
- tl.whereaddressed by · tl.where is a Triton operation that selects values elementwise according to a condition without requiring divergent branch execution.
- predicationaddressed by · Predication is a technique that uses a boolean condition to control whether an instruction executes or affects a result.
- warpproduced by · A warp is a group of GPU threads scheduled and issued together for execution, typically processing the same instruction on different data.
Questions about warp divergence
What is warp divergence?
Warp divergence occurs when threads in a GPU warp follow different control-flow paths, requiring the paths to be executed sequentially.
Which Manning books explain warp divergence?
Grokking GPU Programming and GPU Programming with Triton.
What should I know before learning warp divergence?
Manning books that teach warp divergence usually introduce parallelism, CPU, memory latency and warp first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

