GPU memory trades capacity for speed: the fastest levels hold less, while the largest levels take longer to reach. From fastest to slowest, the levels are registers, shared memory, L1 cache, L2 cache, and global memory. Global-memory requests pass through L2; an L2 miss travels to DRAM. section 11.14 section 11.1
Think of the levels as places to keep data:
- Registers hold a thread’s private variables; shared memory is a fast, small workspace shared by threads in one block. section 3.6
- L1 and L2 caches are managed automatically by hardware. L1 is local to an SM; L2 is shared across the GPU. section 11.1
- Global memory holds large arrays accessible across the grid, but an access that misses L2 must reach DRAM and can be slow. section 3.6 section 11.1
A school for data
Registers are like a note on your own desk, shared memory like a classroom whiteboard, and global memory like a filing cabinet down the hall: each step offers more space but takes longer to reach. section 3.6
Shared memory can help when threads reuse data or need to communicate within a block. Loading data there has a cost, so skip it when data is not reused unless it helps arrange memory accesses. section 11.14
Figure 11.1 The GPU memory hierarchy as a mountain. Each level up is faster but smaller: global memory at the base offers gigabytes at ~400–600 cycles, while registers at the peak respond in ~1 cycle but hold only a handful of values per thread. Everything above the dashed line lives on-chip, close to the cores; global memory sits off-chip in DRAM.

Use the hierarchy diagram to compare how capacity and latency change across levels. section 11.1




