liveBook · Manning

Memory hierarchy

As explained in GPU Programming with Triton

A memory hierarchy is an ordered arrangement of memory levels that trade capacity and access speed to keep frequently needed data near computation.

A GPU can spend hundreds of clock cycles waiting for data from global memory. Its memory hierarchy places smaller, faster storage closer to computation and larger, slower storage farther away. Keeping frequently needed data near computation reduces trips to global memory and helps kernels use available bandwidth. section 1.4 section 3.2

Figure 1.8 The memory hierarchy organized by speed and size. Registers top the hierarchy as the fastest and smallest storage; global memory sits at the bottom as the largest and slowest. Shared memory and the L1 cache appear on adjacent rows to reflect their distinct programming roles; on NVIDIA GPUs from the Volta architecture onwards, they share the same physical on-chip SRAM on each SM and have the same access latency, so the figure separates them by function, not by speed.
CH01 F07 Fartale

The pyramid orders the five levels by size and speed, with shared memory and L1 distinguished by their programming roles. section 1.4

From the smallest and fastest end of the hierarchy:

  • Registers are private to each thread and hold active operands and results for arithmetic. section 1.4
  • Shared memory and L1 cache serve different programming roles. On NVIDIA Volta and later GPUs, they occupy the same on-chip SRAM on each streaming multiprocessor (SM) and have the same access latency. section 1.4
  • L2 cache sits between the on-chip levels and global memory. On the NVIDIA H100, it is shared at the chip level. section 1.4 section 3.2
  • Global memory provides the largest, slowest tier. The H100 uses HBM3 for global memory. section 1.4 section 3.2

The hierarchy works alongside latency hiding: while some threads wait for memory, other ready threads compute. Efficient kernels still need to make good use of bandwidth; concurrency alone does not remove the cost of moving data. section 1.4 section 3.2

Table 1.2 Measured memory access latency by hierarchy level, in GPU clock cycles, on Hopper-generation GPUs
Memory level Latency (cycles)

Registers

~1

Shared memory / L1 cache

29 to 41

L2 cache

258 to 744, since Hopper’s L2 is split into two partitions and the figure depends on which one a request hits and whether it is a hit or a miss

Global memory (HBM)

479 to 656

Measured Hopper latencies illustrate the scale of the differences. The reported L2 range depends on the partition and whether the request hits or misses, so measured ranges need not be strictly ordered. section 1.4

Inspect the target GPU

For a GPU you use, identify its model, architecture, SM count, memory type and capacity, and peak memory bandwidth. Compare these specifications with the H100 rather than assuming its characteristics apply to your device. section 3.6.1 section 3.2

How GPU Programming with Triton covers it

How deep it goes
Describes five memory levels and their placement, and compares measured Hopper access latencies and H100 bandwidth characteristics.
Examples
Uses Hopper latency measurements and H100 memory organization, plus exercises ranking memory levels and inspecting a GPU.
What you can do after
Evaluate GPU memory characteristics and their implications for kernel performance.
How it connects
Connects memory hierarchy to latency hiding, GPU threads, streaming multiprocessors, and memory bandwidth.

Also explained in

Compare how 5 books explain memory hierarchy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.