liveBook · Manning

Quantization

As explained in Quantization and Fast Inference

Quantization is the representation of numerical values such as neural-network weights and activations with reduced-precision formats.

Quantization represents weights and activations with fewer bits, often replacing 16-bit or 32-bit floating point with 8-bit or 4-bit integers. For memory-bound inference, smaller representations reduce the data stored and moved per token. A 16-bit-to-8-bit weight conversion halves raw weight storage; actual latency and energy gains depend on the hardware and runtime. section 1.0 section 1.2 section 12.4

Integer quantization maps values onto a finite, uniformly spaced grid. An affine mapping uses a scale S for grid spacing and a zero point z for the integer representing real zero: q = clip(round(x/S) + z), with reconstruction x̂ = S(q − z). Values between grid points round; values outside the supported range clip. Symmetric and asymmetric schemes make different choices about this mapping. section 1.3 section 2.0 section 2.3

Fixed-point rounding

In Q3.4, the grid spacing is 1/16 = 0.0625. Encoding 0.3 gives round(0.3 × 16) = 5, which reconstructs as 5/16 = 0.3125: an error of +0.0125. Multiplying two encoded values produces a wider integer that must be rescaled; truncating during that step can introduce systematic downward error. section 2.1

For nearest rounding inside the grid, granular error has magnitude at most S/2. Overload error occurs when values are clipped outside the grid and can grow with their distance from the boundary. At a fixed bit-width, widening the range reduces clipping but makes the grid coarser; narrowing it improves resolution but risks losing outliers. section 2.3 section 1.3.2

Figure 2.5 The granular-versus-overload error tradeoff. Widening the quantization grid (increasing the clipping range) reduces overload error—fewer values are clipped—but coarsens the step size, increasing granular error for values inside the range. Tightening the grid does the reverse. The optimal clipping boundary minimizes total distortion and depends on the shape of the data distribution: heavy-tailed distributions push the boundary outward; tightly concentrated distributions pull it inward.

The diagram shows how the clipping boundary trades granular error against overload error. section 2.3

Choose precision and granularity separately for each target. Weights are static at inference, activations vary by input and may contain outliers, and the transformer KV cache is written per token and repeatedly read. A per-tensor scale is simple; per-channel and group-wise scales can preserve more precision, with choices constrained by tensor statistics, accuracy requirements, and hardware. section 3.0

For large language models, LLM.int8() separates outlier dimensions into an FP16 path; GPTQ redistributes weight-rounding error; AWQ rescales channels associated with high-magnitude activations. Validate the chosen format on the target runtime: compare throughput, memory, and quality, and check that execution does not fall back to full precision. section 7.7 section 12.3 section 12.4

How Quantization and Fast Inference covers it

How deep it goes
Derives scale and zero-point mapping, examines rounding and clipping errors, and develops precision and granularity choices.
Examples
Works through Q3.4 encoding of 0.3 and compares FP16 with four-bit AWQ for Qwen2.5-3B-Instruct on an L4.
What it assumes
Assumes PyTorch and basic neural-network familiarity while introducing numerical methods and hardware architecture from zero.
What you can do after
Evaluate and understand quantization configurations, anticipate accuracy risks, and compare deployment memory, throughput, and quality.
How it connects
Connects quantization to fixed-point arithmetic, calibration, attention, and the KV cache.

Also explained in

Compare how 22 books explain quantization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.