14 Warps and occupancy
This chapter covers
- Warps, the 32-thread units of hardware execution
- The single instruction, multiple threads (SIMT) model
- Branch divergence and its cost
- How occupancy determines GPU utilization
- Registers, shared memory, block size, and how to tune each one
You’ve been writing CUDA kernels for thirteen chapters now, launching thousands of threads, each one running its own piece of code. It feels like every thread is running on its own, doing its own thing. But say one day your kernel runs ten times slower than it should and you can’t figure out why.
There is one thing about how the hardware executes those threads that we haven’t covered yet, and knowing it would let you change a few lines and make that same kernel run several times faster. The difference between a sluggish kernel and a fast one often comes down to understanding how threads are really organized.
As we saw in chapter 2, your GPU doesn’t actually run threads one by one. It bundles them into warps of 32 and marches them through your code together, executing the same instruction at the same time. When all 32 agree on which path to take, it runs at full speed. But the moment one thread says “I’ll go left” while the others go right, the entire group slows down.
In this chapter we’ll look at what happens inside the GPU when you run a kernel. Once you see how the hardware executes your threads, those slowdowns will make sense, and you’ll know how to fix them.