Data transfer moves values between memory and computation. In modern inference, reading model data can cost far more energy than the arithmetic performed on it, making memory movement a practical bottleneck. section 1.1
The cost depends on where data is accessed:
- Approximate energy per access rises from 1–5 pJ for L1 cache and 10–30 pJ for L2 to 300–1000 pJ for HBM or off-chip DRAM. section 1.1
- Quantization reduces the number of bits stored and moved: representing values with fewer bits cuts memory footprint and bandwidth demands. section 1.0
Compare transfer and arithmetic costs
An INT8 multiply-add costs roughly 0.2–0.5 pJ, while an HBM access costs roughly 300–1000 pJ. The comparison illustrates why moving data can dominate inference cost even when arithmetic is inexpensive. section 1.1
Figure 1.1 Fetching data from HBM costs roughly 1,700× more energy than an INT8 multiply-add. This gap explains why inference systems are bottlenecked by memory, not compute.

Compare the energy required to fetch data from HBM with the energy for an INT8 multiply-add. section 1.1


