liveBook · Manning

Chapter 1: Facing the Efficiency Wall

What you will learn in chapter 1 of Quantization and Fast Inference

This introduction explains why modern inference is constrained by data movement and why quantization is a practical response.

After this chapter you will be able to

Section by section

Introduction

This introduction explains why modern inference is constrained by data movement and why quantization is a practical response.

Key terms: memory bandwidth, arithmetic throughput, power draw, quantization, weights, activations, data movement, memory footprint

1.1 The cost crisis in memory, latency, and power

The section shows why moving model data, rather than performing arithmetic, drives the memory, latency, and power costs of LLM inference.

Key terms: memory movement, DRAM, HBM, arithmetic, model weights, KV cache, activations, context length, batch size 1, joules per token

1.2 Why quantization is the practical response

The section explains why reducing precision is a practical response to inference costs, why models often tolerate it, and why the right configuration depends on deployment trade-offs.

Key terms: compute density, off-chip memory, memory traffic, architectural changes, kernel tricks, batching, knowledge distillation, quantization, precision, memory bandwidth

1.3 Mapping Floating Point vs Integer

This section introduces quantization as a move between floating-point and integer number representations.

Key terms: floating point, integer arithmetic, scale, uniform spacing, quantization

1.3.1 Floating point's hidden costs

This section explains how floating point’s adaptable precision works and why its arithmetic and memory costs can outweigh its benefits during inference.

Key terms: mantissa, exponent, variable precision, exponent alignment, normalization, rounding, data movement, unused precision

1.3.2 What integers give up, and what they gain

This section explains why integer quantization trades numerical expressiveness for simpler, more efficient computation, and why that tradeoff need not destroy model accuracy.

Key terms: INT8, fixed grid, range, resolution, floating point, fixed-width values, integer ALUs, control logic, throughput, energy per token

1.4 Summary

Where this is used later

Concepts explained in this chapter

Chapter 2: Building Quantization from First Principles →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.