liveBook · Manning

Memory bandwidth

As explained in GPU Programming with Triton

Memory bandwidth is the rate at which a memory system transfers data.

Memory bandwidth is the rate at which data moves between levels of a memory system, measured in bytes per second. Peak memory bandwidth sets a theoretical ceiling on data movement; it is distinct from latency, the time one access takes. section 3.1.3 section 3.2.4

On a GPU, bandwidth varies across the memory hierarchy. For example, the NVIDIA H100 has roughly 3.35 TB/s of global-memory bandwidth, while on-chip storage can provide much higher bandwidth. Data access patterns and the memory level serving a request therefore affect whether compute units stay busy or wait for data. section 3.1 section 3.2.4

Arithmetic intensity—the number of floating-point operations per byte transferred—helps identify bandwidth-limited work. On the cited H100 specifications, peak FP16 Tensor Core throughput and peak memory bandwidth imply about 590 FLOPs per byte. Kernels with much lower arithmetic intensity can be limited by memory bandwidth rather than compute. section 3.1.3 section 3.1.4

A practical contrast

Standalone element-wise operations, reductions, softmax, and normalization are often memory-bound; matrix multiplication and convolution are often compute-bound because they reuse data heavily. section 3.1.4

Figure 3.2 Visualisation of three kernels as Compute Bound, Memory Bound and Optimal through the roofline model
CH03 F02 Fartale

The roofline plot places arithmetic intensity against achievable performance; its memory roof represents the bandwidth limit. section 3.1.5

How GPU Programming with Triton covers it

How deep it goes
Defines bandwidth, distinguishes it from latency, and relates it to arithmetic intensity and roofline limits.
Examples
Compares H100 bandwidth figures and contrasts memory-bound element-wise work with compute-bound matrix multiplication.
What you can do after
Evaluate whether a kernel is memory-bound by comparing its arithmetic intensity with hardware limits and roofline ceilings.
How it connects
Connects memory bandwidth to arithmetic intensity, the memory wall, the roofline model, and the GPU memory hierarchy.

Also explained in

Compare how 4 books explain memory bandwidth →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.