Memory latency is the time a thread waits between requesting data and receiving it. On a GPU, that wait depends on where the data resides: register access takes a cycle, while global-memory access can take hundreds. section 3.2 section 3.2.4
High latency does not necessarily mean low bandwidth. A GPU can keep many memory requests in flight and run other threads while some wait. But register pressure can reduce the number of concurrent threads available to hide latency; spilling registers to memory can add further delays. section 3.2.4 section 3.2.1
Table 3.1 Memory hierarchy relative speed and scope
| Memory Level | Relative Speed | Scope |
Registers |
Fastest (Baseline) |
Per thread |
Shared Memory / L1 Cache |
Fast (10x-30x slower than registers) |
Per block / Per SM |
L2 Cache |
Medium (~5x slower than L1) |
Chip-wide |
Global Memory |
Slow (~15x-20x slower than L1) |
Device-wide |
Compare the relative speeds and scopes of registers, caches, and global memory. section 3.2.4
Figure 3.3 Memory system of an NVIDIA H100 GPU, showing per-SM register file, unified L1/shared memory, constant caches, the chip-level L2 cache, and HBM3 global memory (with PCIe/NVLink connectivity).
Trace the NVIDIA H100 memory hierarchy from per-SM storage to global memory. section 3.2
