liveBook · Manning

Memory bandwidth

As explained in Quantization and Fast Inference

Memory bandwidth is the rate at which data can be transferred between memory and computational components.

Memory bandwidth is the rate at which data moves between memory and computational components. In large-model inference, it can limit token throughput even when arithmetic capacity is available: the hardware must fetch model data before it can use it. Power and data movement can therefore constrain performance before raw compute does. section 1.0 section 1.1

The amount moved per token matters alongside the available bandwidth. A 7B-parameter model can move roughly 15 GB per generated token. Longer contexts add KV-cache traffic: model weight traffic stays fixed while the cache grows with context length. section 1.4 section 1.1

Figure 1.2 Memory traffic per token as context length grows. Model weights (filled) stay constant, but the KV cache (hatched) scales linearly with context. At 128K tokens, total memory traffic reaches 78 GB per token—5.5× more than at 512 tokens.

Compare fixed weight traffic with growing KV-cache traffic: the illustrated total reaches 78 GB per token at 128K context, 5.5 times the traffic at 512 tokens. section 1.1

Quantization reduces the bits used for weights and activations, cutting the data that must be stored and moved. Moving from 16-bit floats to 8-bit or 4-bit integers reduces footprint, bandwidth demand, and energy roughly in proportion to the bit reduction. The gain comes with an approximation tradeoff, so reduced traffic alone does not establish model quality or achieved throughput. section 1.0 section 1.4

Estimating a single-stream bandwidth ceiling

For Llama-2-7B Q4_K_M under llama.cpp, an NVIDIA L4 has 300 GB/s of memory bandwidth. Applying an assumed 65% bandwidth efficiency gives 195 GB/s; dividing by the 3.9 GB model file size gives an estimate of about 50 tokens/s. This is an estimate for the stated workload and efficiency, not a guaranteed rate. section 10.1

Throughput is only part of the deployment decision. In the stated Google Cloud comparison, the L4 instance costs about $3.93 per million tokens at its single-stream ceiling, versus $15.40 for the CPU instance at its own ceiling. Both are billed hourly, so realized cost per token depends on utilization as well as the throughput ceiling. section 10.1

How Quantization and Fast Inference covers it

How deep it goes
Explains traffic per token and gives a bandwidth-based throughput estimate, without deriving memory-system internals.
Examples
Compares a 7B model's per-token traffic and estimates Llama-2-7B Q4_K_M throughput on an NVIDIA L4.
What it assumes
Assumes familiarity with PyTorch and basic neural networks, but not numerical methods or hardware architecture.
What you can do after
Evaluate inference bottlenecks and estimate single-stream throughput and cost using bandwidth, model size, efficiency, and utilization.
How it connects
Connects memory bandwidth to quantization, KV cache, inference throughput, power consumption, and deployment cost.

Also explained in

Compare how 4 books explain memory bandwidth →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.