Memory bandwidth
Memory bandwidth is the rate at which data can be transferred between a memory system and a processor or other computing component.
CUDA for LLMs
Memory bandwidth is the rate at which data can be transferred between memory and a processor.
- How it is used
- Uses memory bandwidth to identify and optimize data-movement bottlenecks in transformer inference kernels.
- How deep it goes
- Traces bandwidth from GEMV arithmetic intensity through coalescing, vectorization, profiling, and saturation against a cuBLAS baseline.
- Examples
- Compares naive and optimized GEMV on an RTX 3090, alongside softmax and LayerNorm profiling results.
- What you can do after
- Diagnose bandwidth bottlenecks with Nsight Compute and compare kernel changes using throughput, occupancy, and runtime.
Quantization and Fast Inference
Memory bandwidth is the rate at which data can be transferred between memory and computational components.
- How it is used
- Uses bandwidth to identify data movement as an inference bottleneck and motivate lower-precision deployment.
- How deep it goes
- Explains traffic per token and gives a bandwidth-based throughput estimate, without deriving memory-system internals.
- Examples
- Compares a 7B model's per-token traffic and estimates Llama-2-7B Q4_K_M throughput on an NVIDIA L4.
- What you can do after
- Evaluate inference bottlenecks and estimate single-stream throughput and cost using bandwidth, model size, efficiency, and utilization.
Read how Quantization and Fast Inference explains memory bandwidth →
GPU Programming with Triton
Memory bandwidth is the rate at which a memory system transfers data.
- How it is used
- Uses peak memory bandwidth to explain GPU bottlenecks and bound achievable kernel performance.
- How deep it goes
- Defines bandwidth, distinguishes it from latency, and relates it to arithmetic intensity and roofline limits.
- Examples
- Compares H100 bandwidth figures and contrasts memory-bound element-wise work with compute-bound matrix multiplication.
- What you can do after
- Evaluate whether a kernel is memory-bound by comparing its arithmetic intensity with hardware limits and roofline ceilings.
Read how GPU Programming with Triton explains memory bandwidth →
Grokking GPU Programming
Memory bandwidth is the rate at which data can be transferred between a processor and memory.
- How it is used
- Frames memory bandwidth as a common bottleneck when diagnosing GPU kernel performance.
- How deep it goes
- Connects bandwidth limits to global-memory access patterns and previews arithmetic intensity without providing worked calculations.
- Examples
- Uses a correct 10,000-thread kernel that barely outperforms a CPU version to illustrate the bottleneck.
- What you can do after
- Understand why a highly parallel GPU kernel can still run slowly when its threads access memory poorly.
Read how Grokking GPU Programming explains memory bandwidth →
Also covered in
- Timeless Algorithms, The Foundational IdeasIn early access (MEAP)
What to know first
- GPU memoryGPU memory is memory available to a graphics processing unit for storing data and state used in graphics or computation.
- throughputThroughput is the rate at which a system processes or completes work, data, or requests over a specified period.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- convolutionConvolution is an operation that applies a filter or kernel across an input, combining values from local neighborhoods to produce output values.
Related concepts
- Nsight Computemeasured by · Nsight Compute is NVIDIA’s interactive profiler for analyzing and optimizing CUDA kernel performance and resource use.
- throughputcontrasts with · Throughput is the rate at which a system processes or completes work, data, or requests over a specified period.
- decode phaseused by · The decode phase is the autoregressive inference stage in which a model generates output tokens sequentially from previously processed context.
- memory latencycontrasts with · Memory latency is the time between requesting data from a memory system and receiving the requested data.
- memory wallcontrasts with · The memory wall is a performance limitation that occurs when data movement between memory and computation is slower than the processor's ability to perform arithmetic.
- parameter countcontrasts with · Parameter count is the number of numerical parameters in a computational model, typically the learned parameters of a machine-learning model.
Questions about memory bandwidth
What is memory bandwidth?
Memory bandwidth is the rate at which data can be transferred between a memory system and a processor or other computing component.
Which Manning books explain memory bandwidth?
CUDA for LLMs, Quantization and Fast Inference, GPU Programming with Triton and Grokking GPU Programming.
What should I know before learning memory bandwidth?
Manning books that teach memory bandwidth usually introduce GPU memory, throughput, attention and convolution first.
What is the difference between memory bandwidth and throughput?
Memory bandwidth is the rate at which data can be transferred between a memory system and a processor or other computing component. Throughput is the rate at which a system processes or completes work, data, or requests over a specified period.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



