liveBook · Manning

Memory hierarchy

As explained in Grokking GPU Programming

A memory hierarchy is an arrangement of storage levels with differing capacities, access latencies, and bandwidths.

GPU memory trades capacity for speed: the fastest levels hold less, while the largest levels take longer to reach. From fastest to slowest, the levels are registers, shared memory, L1 cache, L2 cache, and global memory. Global-memory requests pass through L2; an L2 miss travels to DRAM. section 11.14 section 11.1

Think of the levels as places to keep data:

  • Registers hold a thread’s private variables; shared memory is a fast, small workspace shared by threads in one block. section 3.6
  • L1 and L2 caches are managed automatically by hardware. L1 is local to an SM; L2 is shared across the GPU. section 11.1
  • Global memory holds large arrays accessible across the grid, but an access that misses L2 must reach DRAM and can be slow. section 3.6 section 11.1

A school for data

Registers are like a note on your own desk, shared memory like a classroom whiteboard, and global memory like a filing cabinet down the hall: each step offers more space but takes longer to reach. section 3.6

Shared memory can help when threads reuse data or need to communicate within a block. Loading data there has a cost, so skip it when data is not reused unless it helps arrange memory accesses. section 11.14

Figure 11.1 The GPU memory hierarchy as a mountain. Each level up is faster but smaller: global memory at the base offers gigabytes at ~400–600 cycles, while registers at the peak respond in ~1 cycle but hold only a handful of values per thread. Everything above the dashed line lives on-chip, close to the cores; global memory sits off-chip in DRAM.

Use the hierarchy diagram to compare how capacity and latency change across levels. section 11.1

How Grokking GPU Programming covers it

How deep it goes
Explains five levels, approximate latencies and capacities, L2 access, and shared-memory tradeoffs.
Examples
Uses a school analogy and a mountain diagram to illustrate storage levels and their tradeoffs.
What you can do after
Evaluate whether repeated data access justifies shared memory, and apply cooperative loading and block communication.
How it connects
Connects GPU memory to registers, shared memory, caches, global memory, DRAM, and occupancy.

Also explained in

Compare how 5 books explain memory hierarchy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.