liveBook · Manning

L2 cache

As explained in GPU Programming with Triton

An L2 cache is a hardware-managed GPU cache that is generally larger and farther from compute units than the L1 cache.

When a GPU kernel repeatedly accesses global memory, an L2 cache hit can supply data without another trip to DRAM. L2 is hardware-managed, so the programmer does not explicitly move data into it. It is generally larger and farther from the compute units than L1 cache. section 3.2.3 section 1.4

L1 is private to each streaming multiprocessor (SM), while L2 is shared across the GPU. L2 sits between the SMs and the memory controllers: global-memory reads and writes pass through it, as do host-device transfers over PCIe or NVLink. Its bandwidth can therefore matter alongside DRAM bandwidth. section 3.2.3

Figure 3.3 Memory system of an NVIDIA H100 GPU, showing per-SM register file, unified L1/shared memory, constant caches, the chip-level L2 cache, and HBM3 global memory (with PCIe/NVLink connectivity).
CH03 F03 Fartale

The H100 memory diagram places chip-wide L2 between the per-SM storage and HBM3 global memory. section 3.2

Cache reuse is one way to reduce expensive memory traffic; latency hiding is another. While some threads wait for data, the GPU can run other threads. Kernel scheduling choices can also encourage reuse: grouped program IDs in a matrix multiplication kernel encourage L2 reuse. section 1.4 section 5.10

H100 capacity and bandwidth

The NVIDIA H100 has roughly 50 MB of L2 cache and approximately 5.5 TB/s of L2 bandwidth. These figures are hardware-specific, not universal cache properties. section 3.2.3

Table 1.2 Measured memory access latency by hierarchy level, in GPU clock cycles, on Hopper-generation GPUs
Memory level Latency (cycles)

Registers

~1

Shared memory / L1 cache

29 to 41

L2 cache

258 to 744, since Hopper’s L2 is split into two partitions and the figure depends on which one a request hits and whether it is a hit or a miss

Global memory (HBM)

479 to 656

Hopper latency measurements show that L2 access costs vary with the partition accessed and whether the request hits or misses. section 1.4

How GPU Programming with Triton covers it

How deep it goes
Explains hardware-managed caching, shared placement and traffic paths, with H100 capacity and bandwidth figures.
Examples
Illustrates L2 placement on the H100 and uses grouped program IDs in matrix multiplication to encourage reuse.
What you can do after
Evaluate kernel memory behavior by considering cache reuse and L2 bandwidth alongside DRAM traffic.
How it connects
Connects L2 cache to L1 cache, global memory, DRAM bandwidth and latency hiding.

Also explained in

Compare how 2 books explain L2 cache →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.