liveBook · Manning

Memory bound

As explained in GPU Programming with Triton

A memory-bound regime is a performance regime in which data-transfer capacity limits execution more than arithmetic throughput does.

A GPU kernel is memory-bound when data arrives too slowly to keep its arithmetic units busy. Arithmetic intensity—the operations performed per byte moved—helps distinguish this regime from compute-bound execution. section 3.1.3 section 3.1.4

\[\text{Arithmetic Intensity} = \frac{N_{\text{op}}}{N_{\text{byte}}}.\]

Arithmetic intensity relates a kernel’s computation to its data movement; lower values indicate less work per byte transferred. section 3.1.3

The roofline model plots arithmetic intensity against performance, with compute and memory ceilings. A kernel’s position indicates which resource limits it and how much optimization headroom remains. section 3.1.5

Common memory-bound workloads include:

  • Element-wise operations, reductions, standalone softmax, and normalization, which perform relatively little arithmetic per byte moved. section 3.1.4
  • The elementwise part of SwiGLU: intermediate writes and reads lower arithmetic intensity. A fused Triton kernel keeps activation and multiplication intermediates in registers, avoiding those global-memory transfers. section 3.3.1

How GPU Programming with Triton covers it

How deep it goes
Defines the regime through arithmetic intensity and roofline ceilings, using a first-order model rather than a detailed performance analysis.
Examples
Uses vector addition, reductions, softmax, normalization, and fused SwiGLU to illustrate memory-bound workloads.
What you can do after
Evaluate a kernel’s likely bottleneck using arithmetic intensity and roofline ceilings, then identify data transfers fusion can remove.
How it connects
Connects arithmetic intensity, the roofline model, compute-bound kernels, and kernel fusion.

Also explained in

Compare how 18 books explain memory bound →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.