liveBook · Manning

Memory transfer

As explained in Quantization and Fast Inference

Memory movement is the transfer of data between processors and levels of the memory hierarchy during computation.

Memory transfer moves data between processors and levels of the memory hierarchy during computation. In LLM inference, moving data can cost more energy than arithmetic, making memory traffic a practical bottleneck. section 1.1

Figure 1.1 Fetching data from HBM costs roughly 1,700× more energy than an INT8 multiply-add. This gap explains why inference systems are bottlenecked by memory, not compute.

Compare the energy of fetching data from HBM with an INT8 multiply-add to see why data movement can dominate computation. section 1.1

Figure 1.2 Memory traffic per token as context length grows. Model weights (filled) stay constant, but the KV cache (hatched) scales linearly with context. At 128K tokens, total memory traffic reaches 78 GB per token—5.5× more than at 512 tokens.

See how memory traffic per token grows with context length as the KV cache expands. section 1.1

How Quantization and Fast Inference covers it

How deep it goes
Defines memory movement and compares approximate energy costs for arithmetic and memory access.
Examples
Uses an HBM-versus-INT8 energy comparison and a context-length chart of memory traffic per token.
What you can do after
Understand why moving data can constrain inference more than arithmetic.
How it connects
Connects memory hierarchy, inference, HBM access, and KV-cache traffic.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.