liveBook · Manning

Memory latency

As explained in GPU Programming with Triton

The time between requesting a memory access and receiving its data.

Memory latency is the time a thread waits between requesting data and receiving it. On a GPU, that wait depends on where the data resides: register access takes a cycle, while global-memory access can take hundreds. section 3.2 section 3.2.4

High latency does not necessarily mean low bandwidth. A GPU can keep many memory requests in flight and run other threads while some wait. But register pressure can reduce the number of concurrent threads available to hide latency; spilling registers to memory can add further delays. section 3.2.4 section 3.2.1

Table 3.1 Memory hierarchy relative speed and scope
Memory Level Relative Speed Scope

Registers

Fastest (Baseline)

Per thread

Shared Memory / L1 Cache

Fast (10x-30x slower than registers)

Per block / Per SM

L2 Cache

Medium (~5x slower than L1)

Chip-wide

Global Memory

Slow (~15x-20x slower than L1)

Device-wide

Compare the relative speeds and scopes of registers, caches, and global memory. section 3.2.4

Figure 3.3 Memory system of an NVIDIA H100 GPU, showing per-SM register file, unified L1/shared memory, constant caches, the chip-level L2 cache, and HBM3 global memory (with PCIe/NVLink connectivity).
CH03 F03 Fartale

Trace the NVIDIA H100 memory hierarchy from per-SM storage to global memory. section 3.2

How GPU Programming with Triton covers it

How deep it goes
Traces global-memory access and explains latency hiding and register-pressure trade-offs, with H100 figures but no kernel implementation.
Examples
Compares register and HBM3 access times and illustrates the H100 memory hierarchy.
What you can do after
Evaluate performance trade-offs by relating memory location and register use to latency hiding.
How it connects
Connects memory latency to memory hierarchy, bandwidth, registers, occupancy, and concurrent thread scheduling.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.