liveBook · Manning

L1 cache

As explained in GPU Programming with Triton

An L1 cache is a small, hardware-managed cache close to the GPU compute units.

Repeated global-memory reads can leave GPU threads waiting for data. L1 cache is a small, hardware-managed cache private to each streaming multiprocessor (SM). It holds recently accessed global-memory data so later accesses can hit nearby storage instead of reaching DRAM. section 1.4 section 3.2.3

L1 is part of a hierarchy that also includes registers, shared memory, L2 cache, and global memory. L2 is shared across the GPU, while each SM has its own L1. An L1 hit is generally much faster than an access to L2 or high-bandwidth memory (HBM). section 3.2 section 3.2.3

Figure 1.8 The memory hierarchy organized by speed and size. Registers top the hierarchy as the fastest and smallest storage; global memory sits at the bottom as the largest and slowest. Shared memory and the L1 cache appear on adjacent rows to reflect their distinct programming roles; on NVIDIA GPUs from the Volta architecture onwards, they share the same physical on-chip SRAM on each SM and have the same access latency, so the figure separates them by function, not by speed.
CH01 F07 Fartale

The hierarchy separates L1 and shared memory by programming role, even where they share physical storage and access latency. section 1.4

On NVIDIA GPUs from Volta onward, L1 and shared memory use the same on-chip SRAM and have the same access latency, but serve different roles. Hardware manages L1 automatically; shared memory supports explicit cooperation and data placement. Partitioning the SRAM gives a kernel a trade-off: more shared memory leaves less L1, and vice versa. section 1.4 section 3.2.2 section 3.2.3

H100 storage trade-off

The NVIDIA H100 provides 256 KB of unified L1 and shared-memory storage with different partitioning choices. A kernel that reuses explicitly placed data may benefit from more shared memory; one that benefits more from hardware-managed caching may prefer more L1. In Triton, programmers do not declare shared-memory buffers explicitly: the compiler and backend may use on-chip storage when lowering blocked tensor operations. section 3.2.2

How GPU Programming with Triton covers it

How deep it goes
Explains cache placement, reuse, latency, and the shared-memory partitioning trade-off without detailing cache policies.
Examples
Uses H100 unified storage and a memory-hierarchy ranking exercise to illustrate L1’s place in the GPU.
What you can do after
Evaluate how L1 reuse and shared-memory allocation can affect GPU memory performance.
How it connects
Connects L1 cache to shared memory, L2 cache, global memory, and GPU latency hiding.

Also explained in

Compare how 2 books explain L1 cache →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.