Chapter 14: Warps and occupancy
Introduces warp-based thread execution and its performance implications when threads take different paths.
After this chapter you will be able to
- Warp execution organization. Explain how the GPU organizes threads into warps and executes their instructions together.
- Divergent path performance. Explain how taking different paths within a warp affects its speed compared with all threads taking the same path.
- Warp and block organization. Describe how threads are grouped into warps and how a block’s warps are placed on an SM.
- Warp execution and divergence. Trace the array-doubling example to explain useful lanes, bounds checks, and divergence.
- Warp synchronization and lockstep. Explain why warp-level code must not rely on lockstep and identify the synchronization choices described here.
- Warp scheduling and latency. Explain how an SM switches among resident warps to hide memory latency and why enough ready warps matter.
- Divergent warp execution. Explain how a warp executes different branch paths and illustrate it with the even-odd example.
- Warp boundary of divergence. Distinguish divergence within a warp from different branch choices made by separate warps.
- Serialized divergence cost. Apply the section’s simple timing model to estimate the cost of divergent paths and relate it to path count.
- Predication and select operations. Explain how predication can avoid divergence for short branches and describe its tradeoff.
- Warp uniform branch conditions. Explain why warp-uniform branch conditions avoid divergence and when assigning work by block is practical.
- Sorting data by branch. Describe how sorting by data type can reduce divergence and explain when its cost may be worthwhile.
- Predication and uniform branches. Explain when to rely on predication and when to use a block-level, warp-uniform branch.
- Sorting and specialized kernels. Explain when sorting by type can reduce data-dependent divergence and why separate kernels can improve on sorting alone.
- Divergence and memory efficiency. Explain why divergent paths can make a warp's memory accesses less efficient as well as execute sequentially.
- Occupancy as filled capacity. Explain occupancy as the fraction of an SM’s maximum warp capacity that is filled, and illustrate it with an example from the train analogy.
- Occupancy resource ceilings. Explain how multiple resource limits determine how many warps can reside on an SM.
- Register file residency limit. Explain how registers per thread limit residency, and apply the relationship to the section’s occupancy exercise.
- Shared memory residency limit. Explain how shared memory reserved per block limits the blocks and warps that can reside on an SM.
- Block size and slots. Explain how block size interacts with block slots and per-block resources, including the benefit of choosing a multiple of 32 threads.
- Occupancy and measured performance. Explain why higher occupancy does not necessarily produce better performance, using the register-use example.
- Workload bottlenecks and occupancy. Distinguish how occupancy can help memory-bound and compute-bound kernels, and describe how arithmetic intensity helps identify a likely memory bottleneck.
- Occupancy and runtime measurement. Describe how to measure achieved occupancy and choose a block size based on performance rather than occupancy alone.
- Launch bounds register guidance. Explain what the two values in the section's __launch_bounds__ example tell the compiler and what they do not guarantee.
- Occupancy constraint calculation. Calculate the example’s occupancy by comparing the register, shared-memory, and block limits.
- Warp execution order assumptions. Explain why assuming warps run in a fixed order within a block is unsafe.
- Resident warps and speed. Explain why more resident warps do not always make a kernel faster.
- Cross-warp execution race. Explain why a thread reading a value written by another warp can see a stale value and produce erratic results.
- Block-wide synchronization and visibility. Describe where to place __syncthreads() to fix a cross-warp read and explain its barrier and memory-visibility effects within a block.
- Occupancy and register spilling. Explain how forcing high occupancy can make a kernel slower despite having more resident warps.
- Measured occupancy tradeoffs. Describe how to choose between kernel configurations when occupancy and per-thread speed trade off.
What it builds on
- How the hardware runs your threads: warps and SMs (chapter 2) · The earlier introduction to warps and hardware thread execution.
- The three-level hierarchy: grids, blocks, and threads (chapter 3) · The organization of threads into blocks.
- Bounds check (chapter 2) · A bounds check prevents threads from accessing elements outside the array.
- Your first multi-block kernel (chapter 3) · The multi-block array-doubling kernel.
- The barrier: __syncthreads() (chapter 5) · The block barrier __syncthreads().
- The memory bottleneck (chapter 10) · Memory access can be a performance bottleneck.
- The global index formula (chapter 3) · The global index combines a block’s position with a thread’s index within the block.
- Tree reduction: from concept to code (chapter 5) · Tree reduction and its pairing of threads across successive rounds.
Section by section
Introduction
Introduces warp-based thread execution and its performance implications when threads take different paths.
- Warp execution organization: Explain how the GPU organizes threads into warps and executes their instructions together.
Why it matters: This organization helps explain why a kernel may behave differently from what independent thread execution suggests. - Divergent path performance: Explain how taking different paths within a warp affects its speed compared with all threads taking the same path.
Why it matters: Different choices by threads in the same warp can make a kernel slower.
Key terms: warps, single instruction, multiple threads, SIMT, branch divergence, warp
14.1 The marching band inside your GPU
This section explains how NVIDIA warps execute thread code, synchronize safely, and use scheduling to hide memory latency.
- Warp and block organization: Describe how threads are grouped into warps and how a block’s warps are placed on an SM.
Why it matters: Warp grouping connects the thread blocks you launch to the units the hardware executes. - SIMT and SIMD distinction: Distinguish SIMT thread programming from explicit SIMD programming as described in the section.
Why it matters: This distinction explains why CUDA code can look like ordinary single-thread code even though the hardware executes warps. - Warp execution and divergence: Trace the array-doubling example to explain useful lanes, bounds checks, and divergence.
Why it matters: A bounds check can keep accesses safe while leaving part of a warp idle. - Warp synchronization and lockstep: Explain why warp-level code must not rely on lockstep and identify the synchronization choices described here.
Why it matters: Issuing instructions to a warp does not by itself make thread-to-thread communication safe. - Warp scheduling and latency: Explain how an SM switches among resident warps to hide memory latency and why enough ready warps matter.
Why it matters: Having other warps ready to run helps the GPU do useful work while memory requests are pending.
Key terms: warp, block, streaming multiprocessor, SM, SIMT, SIMD, branch divergence, lane, bounds check, lockstep
14.2 When the band splits: branch divergence
This section explains how branch divergence serializes paths within a warp and how to estimate its cost.
- Divergent warp execution: Explain how a warp executes different branch paths and illustrate it with the even-odd example.
Why it matters: This explains why a branch can leave part of a warp idle. - Warp boundary of divergence: Distinguish divergence within a warp from different branch choices made by separate warps.
Why it matters: It helps you identify which branch differences actually cause divergence. - Serialized divergence cost: Apply the section’s simple timing model to estimate the cost of divergent paths and relate it to path count.
Why it matters: This model lets you estimate the cost of a warp split without assuming the paths overlap. - Even-odd index comparison: Explain why the example’s global-index and block-local-index parity tests produce the same warp split for 256-thread blocks.
Why it matters: It shows how to reason about the actual branch split rather than just the name of the index used.
Key terms: warp, branch divergence, masking, Golden Rule of Divergence, serialized model, utilization, slowdown, global index, threadIdx.x, idx % 2
14.3 Three strategies for managing divergence
This section explains three strategies for managing warp divergence and illustrates warp-uniform work with tree reduction.
- Predication and select operations: Explain how predication can avoid divergence for short branches and describe its tradeoff.
Why it matters: Predication can replace the overhead of a split warp with a small amount of extra arithmetic. - Warp uniform branch conditions: Explain why warp-uniform branch conditions avoid divergence and when assigning work by block is practical.
Why it matters: Organizing work so a whole warp makes the same choice avoids executing different paths within that warp. - Stride based reduction pairing: Explain how stride-based pairing in tree reduction reduces divergence compared with neighbor pairing.
Why it matters: This shows how the placement of active threads can matter as much as their number. - Sorting data by branch: Describe how sorting by data type can reduce divergence and explain when its cost may be worthwhile.
Why it matters: Sorting can make data-dependent branches more uniform, but its overhead must be justified.
Key terms: predication, select op, branchless code, divergence, warp-uniform, blockIdx.x, threadIdx.x, tree reduction, stride-based pairing, neighbor pairing
14.4 Which strategy should you choose?
This section explains how to choose a divergence-management strategy and why divergence also reduces memory efficiency.
- Predication and uniform branches: Explain when to rely on predication and when to use a block-level, warp-uniform branch.
Why it matters: The right choice depends on whether the branch is cheap or the work can be divided uniformly by block. - Sorting and specialized kernels: Explain when sorting by type can reduce data-dependent divergence and why separate kernels can improve on sorting alone.
Why it matters: Grouping data can reduce divergent work, but its cost and remaining boundary effects matter. - Divergence and memory efficiency: Explain why divergent paths can make a warp's memory accesses less efficient as well as execute sequentially.
Why it matters: Divergence can waste both execution time and the benefit of bundling a warp's memory accesses.
Key terms: predication, branchless select, warp-uniform branches, blockIdx, threadIdx, data-dependent, sort first, separate kernels per type, divergence, divergent warp
14.5 Filling the stage: occupancy
This section defines SM occupancy using a train analogy and shows how to express it as a percentage of the maximum warps.
- Occupancy as filled capacity: Explain occupancy as the fraction of an SM’s maximum warp capacity that is filled, and illustrate it with an example from the train analogy.
Why it matters: Occupancy gives you a way to describe how much of an SM’s warp capacity is in use. - Architecture dependent warp capacity: Describe how an SM’s maximum warp capacity varies by GPU architecture and where to check it.
Why it matters: The correct maximum is needed to interpret an occupancy percentage for a particular GPU.
Key terms: occupancy, SM, warp, max warps, compute capability table
14.6 What limits occupancy: three resources
This section explains how registers, shared memory, and block slots limit occupancy on a GPU’s streaming multiprocessors.
- Occupancy resource ceilings: Explain how multiple resource limits determine how many warps can reside on an SM.
Why it matters: This helps you identify the constraint that actually prevents more warps from residing on an SM. - Register file residency limit: Explain how registers per thread limit residency, and apply the relationship to the section’s occupancy exercise.
Why it matters: Per-thread register use can determine how much parallel work an SM can keep resident. - Register spilling trade-off: Explain why forcing lower register use can increase occupancy yet slow individual threads, and distinguish GPU local memory from physically nearby storage.
Why it matters: Reducing register use is not automatically a performance improvement. - Shared memory residency limit: Explain how shared memory reserved per block limits the blocks and warps that can reside on an SM.
Why it matters: A block’s shared memory allocation can limit occupancy even when other resources have space. - Block size and slots: Explain how block size interacts with block slots and per-block resources, including the benefit of choosing a multiple of 32 threads.
Why it matters: Block size affects both how fully warp slots are used and whether blocks fit within resource limits.
Key terms: occupancy, SM, registers, shared memory, block slots, register file, registers per thread, resident threads, register spilling, local memory
14.7 Occupancy is a means, not a goal
This section explains why occupancy is a performance aid rather than a target and presents ways to measure and tune it.
- Occupancy and measured performance: Explain why higher occupancy does not necessarily produce better performance, using the register-use example.
Why it matters: The fastest kernel is not necessarily the one with the most active warps. - Workload bottlenecks and occupancy: Distinguish how occupancy can help memory-bound and compute-bound kernels, and describe how arithmetic intensity helps identify a likely memory bottleneck.
Why it matters: Knowing the bottleneck helps you decide whether more resident warps are likely to be useful. - Occupancy and runtime measurement: Describe how to measure achieved occupancy and choose a block size based on performance rather than occupancy alone.
Why it matters: Measurement lets you distinguish an occupancy change from an actual speedup. - Launch bounds register guidance: Explain what the two values in the section's __launch_bounds__ example tell the compiler and what they do not guarantee.
Why it matters: Launch bounds can influence register use, but they do not override all other occupancy limits. - Shared memory reuse trade-off: Explain how reducing shared memory per block can affect occupancy and data reuse.
Why it matters: Reducing shared memory can create more scheduling opportunities, but it also sacrifices reuse.
Key terms: occupancy, registers, local memory, throughput, memory-bound, compute-bound, arithmetic intensity, ridge point, resident warps, achieved occupancy
14.8 Worked example: checking all three constraints
This section works through an occupancy calculation, identifies the limiting resource, and explains practical qualifications and trade-offs.
- Occupancy constraint calculation: Calculate the example’s occupancy by comparing the register, shared-memory, and block limits.
Why it matters: Comparing all the constraints tells you which resource actually limits occupancy. - Register reduction trade-off: Explain the proposed launch-bounds adjustment and why its performance should be checked.
Why it matters: Higher occupancy is not necessarily worth the cost of reducing register use. - Register allocation rounding: Explain why actual occupancy can be lower than the simple hand calculation suggests.
Why it matters: Allocation rounding can make a hand estimate overstate the occupancy your GPU can achieve.
Key terms: occupancy, registers, shared memory, block limit, warps, __launch_bounds__, spilling, profile, register allocation unit, occupancy API
14.9 Common mistakes
This section identifies mistaken assumptions about warp execution order and occupancy that can cause failures or slower kernels.
- Warp execution order assumptions: Explain why assuming warps run in a fixed order within a block is unsafe.
Why it matters: Depending on an order the hardware does not guarantee can make a kernel fail intermittently. - Resident warps and speed: Explain why more resident warps do not always make a kernel faster.
Why it matters: Trying to increase resident warps on the assumption that it always helps can slow a kernel down.
Key terms: warps, block, execution order, resident warps, occupancy
14.10 Mistake 1: Assuming cross-warp execution order
This section explains why unsynchronized communication between warps can produce a race and how a block-wide barrier fixes it.
- Cross-warp execution race: Explain why a thread reading a value written by another warp can see a stale value and produce erratic results.
Why it matters: Recognizing this race helps explain why a kernel can work intermittently despite using the same code. - Block-wide synchronization and visibility: Describe where to place __syncthreads() to fix a cross-warp read and explain its barrier and memory-visibility effects within a block.
Why it matters: The barrier gives communicating warps in the same block a defined point at which earlier writes become visible.
Key terms: warp, cross-warp race, stale value, __syncthreads(), barrier, memory fence, block
14.11 Mistake 2: Blindly maximizing occupancy
This section explains why maximizing occupancy can slow a kernel and recommends measuring configurations instead.
- Occupancy and register spilling: Explain how forcing high occupancy can make a kernel slower despite having more resident warps.
Why it matters: More resident warps do not guarantee a faster kernel if each thread becomes slower. - Measured occupancy tradeoffs: Describe how to choose between kernel configurations when occupancy and per-thread speed trade off.
Why it matters: Timing the alternatives helps you choose the configuration that actually runs fastest.
Key terms: occupancy, registers, spill, local memory, warps, configuration
14.12 Summary
Concepts explained in this chapter
- occupancyGPU occupancy is the ratio of active warps on a streaming multiprocessor to the maximum number of warps it can support.
- warp divergenceBranch divergence is a GPU execution inefficiency that occurs when threads in the same warp take different control-flow paths.
- warpA warp is a GPU execution group whose threads typically execute the same instruction concurrently on different data.
- register spillingRegister spilling is the compiler's placement of register values in slower memory when register demand exceeds the available or permitted registers.
- SIMTSIMT is a GPU execution model in which groups of threads execute instructions together while operating on separate data.
- predicationPredication is a technique for avoiding divergent control flow by conditionally executing instructions or computing alternatives and selecting a result.
- warp-uniform branchA warp-uniform branch is a conditional branch for which all threads in a warp take the same path.
- warp schedulerWarp scheduling is the GPU hardware process of selecting warps for execution on available processing resources.
- register fileA register file is a collection of fast on-chip registers that stores values for a processor's active threads.
- hardware registerA GPU register is a tiny, fast storage location private to an individual thread.
- select instructionA select instruction chooses one of two values based on a condition without splitting warp control flow.
- __syncthreads()__syncthreads is a CUDA barrier operation that makes threads in one block wait until all threads in that block reach the barrier.
- barrierA barrier is a synchronization mechanism that makes participating threads wait until the required threads reach a specified point.
- __launch_bounds____launch_bounds__ is an NVIDIA-specific CUDA compiler attribute that specifies a kernel's maximum threads per block and optionally a minimum number of blocks per multiprocessor.
- SIMD hardwareA SIMD/AVX unit is processor hardware that applies one vector instruction to multiple data elements at once.
- block sizeThreads per block is the configured number of GPU threads assigned to each CUDA thread block in a kernel launch.
- arithmetic intensityArithmetic intensity is the number of floating-point operations performed per byte transferred to and from memory.
- compute-boundCompute-bound describes a computation whose performance is limited primarily by the processor's computational capacity.
- memory boundA memory-bound kernel is a GPU kernel whose performance is limited primarily by memory data transfer.
- shared memoryShared memory is a fast, explicitly managed memory space accessible to threads within a GPU thread block.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
