10 Memory access patterns
This chapter covers
- Why memory bandwidth is usually the bottleneck
- Calculating arithmetic intensity
- The Snapshot Rule for analyzing access patterns
- Coalesced, strided, and scattered access patterns
- Optimizing data layouts for parallel performance
You’ve just finished writing a new kernel. You launch 10,000 threads, sit back, and wait for a big speedup. But the result is barely faster than the CPU version. The algorithm is correct, and the thread organization is fine. What’s holding the kernel back is how the threads are reading memory.
This chapter is about that invisible bottleneck. You’ll learn how GPUs actually move data, why the pattern of memory access matters more than the amount of computation, and how small changes in your code can make it 3 to 5 times faster. Chapter 3 gave a brief tour of the memory hierarchy (registers, shared memory, and global memory), and since then we’ve written real Map, Reduce, Scan, Histogram, and Stencil kernels. That makes this the right moment to look at why those kernels run the way they do. The answer lies in how they access global memory, and understanding that access pattern lays the groundwork for the optimizations in the chapters ahead.
The memory bottleneck
Consider an 8-lane highway with a single toll gate at the entrance: however wide the road, every car has to squeeze through the one gate, so the gate sets the pace.