liveBook · Manning

Shared memory

As explained in GPU Programming with Triton

Shared memory is explicitly managed on-chip GPU memory visible to all threads in a single block.

Shared memory is programmable on-chip GPU memory accessible to all threads in a thread block. It gives those threads a place to share data and cooperate, unlike registers, which are private to individual threads. It sits on each streaming multiprocessor (SM). section 3.2.2

The performance motivation is the memory wall: global memory is much slower than on-chip storage, so arithmetic units can spend time waiting for data. Shared memory can stage a value once for reuse by multiple threads rather than having each thread fetch it from global memory. Threads must synchronize before consuming data another thread has staged. section 3.1 section 3.2.2

Sharing matrix tiles

In matrix multiplication, threads in a block load tiles of the input matrices into shared memory, then use the shared data to compute partial results. Shared memory can also change an access pattern: threads load data from global memory in one pattern and access it from shared memory in another. section 3.2.2

Shared memory and L1 cache have different programming roles: shared memory is software-managed, while the cache is hardware-managed. On NVIDIA GPUs from Volta onward, they occupy the same on-chip SRAM and have the same access latency. Choosing more shared memory can leave less capacity for L1 caching, so the useful partition depends on the kernel. section 1.4 section 3.2.2

Figure 1.8 The memory hierarchy organized by speed and size. Registers top the hierarchy as the fastest and smallest storage; global memory sits at the bottom as the largest and slowest. Shared memory and the L1 cache appear on adjacent rows to reflect their distinct programming roles; on NVIDIA GPUs from the Volta architecture onwards, they share the same physical on-chip SRAM on each SM and have the same access latency, so the figure separates them by function, not by speed.
CH01 F07 Fartale

Compare shared memory with registers, caches, and global memory by role and position in the GPU memory hierarchy. section 1.4

Triton programmers do not explicitly declare shared-memory buffers. The compiler and backend lower blocked tensor operations into code that may use on-chip storage, depending on the kernel and target. Reductions and scans can require partial results to be exchanged between warps within a block. section 3.2.2

How GPU Programming with Triton covers it

How deep it goes
Explains staging, synchronization, access-pattern changes, and cache partitioning, while leaving Triton storage decisions to compiler lowering.
Examples
Uses matrix multiplication tiles and inter-warp partial-result exchange in reductions and scans.
What you can do after
Evaluate/understand when shared storage can reduce global-memory reads and how its capacity competes with L1 cache.
How it connects
Connects thread blocks, registers, global memory, L1 cache, reductions, and scans.

Also explained in

Compare how 5 books explain shared memory →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.