Memory bandwidth is the rate at which data moves between levels of a memory system, measured in bytes per second. Peak memory bandwidth sets a theoretical ceiling on data movement; it is distinct from latency, the time one access takes. section 3.1.3 section 3.2.4
On a GPU, bandwidth varies across the memory hierarchy. For example, the NVIDIA H100 has roughly 3.35 TB/s of global-memory bandwidth, while on-chip storage can provide much higher bandwidth. Data access patterns and the memory level serving a request therefore affect whether compute units stay busy or wait for data. section 3.1 section 3.2.4
Arithmetic intensity—the number of floating-point operations per byte transferred—helps identify bandwidth-limited work. On the cited H100 specifications, peak FP16 Tensor Core throughput and peak memory bandwidth imply about 590 FLOPs per byte. Kernels with much lower arithmetic intensity can be limited by memory bandwidth rather than compute. section 3.1.3 section 3.1.4
A practical contrast
Standalone element-wise operations, reductions, softmax, and normalization are often memory-bound; matrix multiplication and convolution are often compute-bound because they reuse data heavily. section 3.1.4
Figure 3.2 Visualisation of three kernels as Compute Bound, Memory Bound and Optimal through the roofline model
The roofline plot places arithmetic intensity against achievable performance; its memory roof represents the bandwidth limit. section 3.1.5



