chapter ten

10 Memory access patterns

 

This chapter covers

  • Why memory bandwidth is usually the bottleneck
  • Calculating arithmetic intensity
  • The Snapshot Rule for analyzing access patterns
  • Coalesced, strided, and scattered access patterns
  • Optimizing data layouts for parallel performance

You’ve just finished writing a new kernel. You launch 10,000 threads, sit back, and wait for a big speedup. But the result is barely faster than the CPU version. The algorithm is correct, and the thread organization is fine. What’s holding the kernel back is how the threads are reading memory.

This chapter is about that invisible bottleneck. You’ll learn how GPUs actually move data, why the pattern of memory access matters more than the amount of computation, and how small changes in your code can make it 3 to 5 times faster. Chapter 3 gave a brief tour of the memory hierarchy (registers, shared memory, and global memory), and since then we’ve written real Map, Reduce, Scan, Histogram, and Stencil kernels. That makes this the right moment to look at why those kernels run the way they do. The answer lies in how they access global memory, and understanding that access pattern lays the groundwork for the optimizations in the chapters ahead.

The memory bottleneck

Consider an 8-lane highway with a single toll gate at the entrance: however wide the road, every car has to squeeze through the one gate, so the gate sets the pace.

Compute-bound or memory-bound?

Arithmetic intensity

How GPUs really read memory

Coalesced access: the ideal case

Strided access: skipping elements

Scattered access: the worst case

Real-world access patterns

AoS and SoA

Matrices

Common mistakes

Mistake 1: "My kernel does a lot of math, so it must be fast"

Mistake 2: Ignoring memory alignment

Summary