The memory wall is the gap between a GPU’s compute throughput and the memory bandwidth available to supply data. When data arrives too slowly, arithmetic units wait idle, limiting performance even when a workload offers plenty of computation. section 3.1 section 3.1.3
Arithmetic intensity—the operations performed per byte moved—helps identify the bottleneck. Low intensity can leave a kernel memory-bound; higher intensity may make it compute-bound. The roofline model relates intensity to hardware compute and memory ceilings. section 3.1.3 section 3.7
Use this ratio to reason about how much computation a kernel performs for each byte it moves. section 3.1.3
Kernel fusion can reduce the data movement that contributes to the memory wall. Separate kernels send intermediate results through global memory; a fused kernel can keep those values in registers instead. The benefit comes from avoiding costly global-memory round trips. section 3.2.4 section 3.7
Fusing chained operations
Fusing SiLU and multiplication reduces memory traffic from 20 to 12 bytes per element, corresponding to a theoretical 1.67× speedup. SwiGLU and Flash Attention illustrate the same fusion principle at larger scales. section 3.7
