liveBook · Manning

Memory bandwidth

Memory bandwidth is the rate at which data can be transferred between a memory system and a processor or other computing component.

How 4 Manning books explain it

CUDA for LLMs

Memory bandwidth is the rate at which data can be transferred between memory and a processor.

How it is used
Uses memory bandwidth to identify and optimize data-movement bottlenecks in transformer inference kernels.
How deep it goes
Traces bandwidth from GEMV arithmetic intensity through coalescing, vectorization, profiling, and saturation against a cuBLAS baseline.
Examples
Compares naive and optimized GEMV on an RTX 3090, alongside softmax and LayerNorm profiling results.
What you can do after
Diagnose bandwidth bottlenecks with Nsight Compute and compare kernel changes using throughput, occupancy, and runtime.

Read how CUDA for LLMs explains memory bandwidth →

Quantization and Fast Inference

Memory bandwidth is the rate at which data can be transferred between memory and computational components.

How it is used
Uses bandwidth to identify data movement as an inference bottleneck and motivate lower-precision deployment.
How deep it goes
Explains traffic per token and gives a bandwidth-based throughput estimate, without deriving memory-system internals.
Examples
Compares a 7B model's per-token traffic and estimates Llama-2-7B Q4_K_M throughput on an NVIDIA L4.
What you can do after
Evaluate inference bottlenecks and estimate single-stream throughput and cost using bandwidth, model size, efficiency, and utilization.

Read how Quantization and Fast Inference explains memory bandwidth →

GPU Programming with Triton

Memory bandwidth is the rate at which a memory system transfers data.

How it is used
Uses peak memory bandwidth to explain GPU bottlenecks and bound achievable kernel performance.
How deep it goes
Defines bandwidth, distinguishes it from latency, and relates it to arithmetic intensity and roofline limits.
Examples
Compares H100 bandwidth figures and contrasts memory-bound element-wise work with compute-bound matrix multiplication.
What you can do after
Evaluate whether a kernel is memory-bound by comparing its arithmetic intensity with hardware limits and roofline ceilings.

Read how GPU Programming with Triton explains memory bandwidth →

Grokking GPU Programming

Memory bandwidth is the rate at which data can be transferred between a processor and memory.

How it is used
Frames memory bandwidth as a common bottleneck when diagnosing GPU kernel performance.
How deep it goes
Connects bandwidth limits to global-memory access patterns and previews arithmetic intensity without providing worked calculations.
Examples
Uses a correct 10,000-thread kernel that barely outperforms a CPU version to illustrate the bottleneck.
What you can do after
Understand why a highly parallel GPU kernel can still run slowly when its threads access memory poorly.

Read how Grokking GPU Programming explains memory bandwidth →

Also covered in

What to know first

Related concepts

Questions about memory bandwidth

What is memory bandwidth?

Memory bandwidth is the rate at which data can be transferred between a memory system and a processor or other computing component.

Which Manning books explain memory bandwidth?

CUDA for LLMs, Quantization and Fast Inference, GPU Programming with Triton and Grokking GPU Programming.

What should I know before learning memory bandwidth?

Manning books that teach memory bandwidth usually introduce GPU memory, throughput, attention and convolution first.

What is the difference between memory bandwidth and throughput?

Memory bandwidth is the rate at which data can be transferred between a memory system and a processor or other computing component. Throughput is the rate at which a system processes or completes work, data, or requests over a specified period.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.