liveBook · Manning

Data transfer

As explained in Quantization and Fast Inference

Data movement is the transfer of data among storage, memory, and computational locations during computation.

Data transfer moves values between memory and computation. In modern inference, reading model data can cost far more energy than the arithmetic performed on it, making memory movement a practical bottleneck. section 1.1

The cost depends on where data is accessed:

  • Approximate energy per access rises from 1–5 pJ for L1 cache and 10–30 pJ for L2 to 300–1000 pJ for HBM or off-chip DRAM. section 1.1
  • Quantization reduces the number of bits stored and moved: representing values with fewer bits cuts memory footprint and bandwidth demands. section 1.0

Compare transfer and arithmetic costs

An INT8 multiply-add costs roughly 0.2–0.5 pJ, while an HBM access costs roughly 300–1000 pJ. The comparison illustrates why moving data can dominate inference cost even when arithmetic is inexpensive. section 1.1

Figure 1.1 Fetching data from HBM costs roughly 1,700× more energy than an INT8 multiply-add. This gap explains why inference systems are bottlenecked by memory, not compute.

Compare the energy required to fetch data from HBM with the energy for an INT8 multiply-add. section 1.1

How Quantization and Fast Inference covers it

How deep it goes
Quantifies energy across memory levels and arithmetic, without detailing transfer mechanisms or runtime implementations.
Examples
Compares approximate energy costs for L1, L2, HBM, and INT8 multiply-add operations.
What it assumes
Assumes familiarity with basic neural networks, while requiring no prior hardware-architecture or numerical-methods background.
What you can do after
Understand why memory movement can constrain inference more than arithmetic.
How it connects
Connects data movement to quantization, memory hierarchy, and inference.

Also explained in

Compare how 3 books explain data transfer →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.