Memory transfer moves data between processors and levels of the memory hierarchy during computation. In LLM inference, moving data can cost more energy than arithmetic, making memory traffic a practical bottleneck. section 1.1
Figure 1.1 Fetching data from HBM costs roughly 1,700× more energy than an INT8 multiply-add. This gap explains why inference systems are bottlenecked by memory, not compute.

Compare the energy of fetching data from HBM with an INT8 multiply-add to see why data movement can dominate computation. section 1.1
Figure 1.2 Memory traffic per token as context length grows. Model weights (filled) stay constant, but the KV cache (hatched) scales linearly with context. At 128K tokens, total memory traffic reaches 78 GB per token—5.5× more than at 512 tokens.

See how memory traffic per token grows with context length as the KV cache expands. section 1.1
