liveBook · Manning

Memory wall

As explained in GPU Programming with Triton

The memory wall is a performance limitation that occurs when data movement between memory and computation is slower than the processor's ability to perform arithmetic.

The memory wall is the gap between a GPU’s compute throughput and the memory bandwidth available to supply data. When data arrives too slowly, arithmetic units wait idle, limiting performance even when a workload offers plenty of computation. section 3.1 section 3.1.3

Arithmetic intensity—the operations performed per byte moved—helps identify the bottleneck. Low intensity can leave a kernel memory-bound; higher intensity may make it compute-bound. The roofline model relates intensity to hardware compute and memory ceilings. section 3.1.3 section 3.7

\[\text{Arithmetic Intensity} = \frac{N_{\text{op}}}{N_{\text{byte}}}.\]

Use this ratio to reason about how much computation a kernel performs for each byte it moves. section 3.1.3

Kernel fusion can reduce the data movement that contributes to the memory wall. Separate kernels send intermediate results through global memory; a fused kernel can keep those values in registers instead. The benefit comes from avoiding costly global-memory round trips. section 3.2.4 section 3.7

Fusing chained operations

Fusing SiLU and multiplication reduces memory traffic from 20 to 12 bytes per element, corresponding to a theoretical 1.67× speedup. SwiGLU and Flash Attention illustrate the same fusion principle at larger scales. section 3.7

How GPU Programming with Triton covers it

How deep it goes
Quantifies the bottleneck with arithmetic intensity and relates it to compute-bound and memory-bound kernels and roofline ceilings.
Examples
Uses SiLU and multiplication, SwiGLU, and Flash Attention to illustrate how fusion reduces global-memory transfers.
What you can do after
Diagnose whether a kernel is compute-bound or memory-bound using arithmetic intensity and relate its performance to hardware ceilings.
How it connects
Connects arithmetic intensity and the roofline model to compute-bound kernels, memory-bound kernels, and kernel fusion.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.