liveBook · Manning

Occupancy

As explained in Grokking GPU Programming

GPU occupancy is the ratio of active warps on a streaming multiprocessor to the maximum number of warps it can support.

A GPU kernel can have thousands of threads but still leave room for more work on each streaming multiprocessor (SM). Occupancy measures how full that room is: it is the number of active warps divided by the maximum number of warps the SM can hold. On NVIDIA GPUs, a warp contains 32 threads. section 14.5 section 14.1

Counting occupied slots

If an SM can hold 64 warps and has 16 active warps, occupancy is 16 ÷ 64, or 25%. With 48 active warps, it is 75%. The maximum depends on the GPU architecture, so check the device’s compute capability table. section 14.5

Occupancy matters because an SM can switch to another resident warp when one stalls waiting for memory. More resident warps can help hide that wait, but a full SM is not automatically a fast one. section 14.12 section 14.7

Three constraints determine how many warps can reside on an SM:

  • Registers per thread: each resident thread needs a share of the SM’s register budget. Higher register use can leave room for fewer blocks. section 14.6
  • Shared memory per block: each resident block reserves part of the SM’s shared memory budget. Larger reservations can reduce the number of blocks that fit. section 14.6
  • Block size and block slots: a block contains whole warps, and an SM can host only a limited number of blocks. Very small blocks can exhaust those slots before filling the available warp capacity. section 14.6

Find the tightest constraint

Suppose an SM has a 64-warp maximum, 65,536 registers, 48 KB of shared memory, and 16 block slots. A kernel uses 256 threads, 48 registers per thread, and 4 KB of shared memory per block. Registers fit five whole blocks, or 40 warps; shared memory and block slots permit more. Registers therefore limit occupancy to 40 ÷ 64, or 62.5%. Actual allocations use architecture-specific chunks, so a hand calculation can overestimate occupancy. section 14.8

A common mistake is to force maximum occupancy and expect a speedup. Reducing the register budget can cause variables to spill into slower local memory, making each thread slower. A memory-bound kernel may benefit from more warps, but it may already have enough to saturate memory bandwidth. Try different block sizes, register budgets, or shared memory sizes, and keep the configuration that actually runs fastest. section 14.11 section 14.7

On a machine with an NVIDIA GPU, Nsight Compute can report achieved occupancy for each kernel. For example, run: ncu --metrics sm__warps_active.avg.pct_of_peak_sustained_active ./your_program. Compare that report and the kernel’s performance across configurations; do not optimize the occupancy percentage alone. section 14.7

How Grokking GPU Programming covers it

How deep it goes
Calculates occupancy from resource constraints, discusses allocation rounding, and explains why higher occupancy can hurt performance.
Examples
Uses a train analogy, numerical occupancy examples, and a worked calculation with register and shared-memory limits.
What it assumes
Assumes familiarity with CUDA kernels and thread launches, then explains occupancy through warps and SM resources.
What you can do after
Diagnose GPU occupancy limits and compare kernel configurations using Nsight Compute and performance measurements.
How it connects
Connects occupancy to warps, latency hiding, registers, shared memory, block size, and register spilling.

Also explained in

Compare how 3 books explain occupancy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.