Chapter 2: Building Quantization from First Principles
This chapter introduction frames quantization as the translation from floating-point neural-network values to the small integers used for inference.
After this chapter you will be able to
- Floating-point to integer translation. Explain that quantization maps continuous floating-point neural-network values onto a finite grid of integer levels for deployment, where inference hardware uses small integer containers.
- Q-format integer interpretation. Explain how a signed Q3.4 value stores an integer while interpreting it with a scale of 1/16, and use that rule to encode or decode a value.
- Fixed-point multiplication and rounding. Trace why multiplying two Q3.4 raw integers produces an extra-scaled result, how a right shift restores Q3.4, and why adding a rounding bias before the shift can reduce systematic truncation drift.
- Fixed-grid range and precision. Explain the range-versus-precision tradeoff when allocating a fixed number of bits, and why measured activation ranges help choose a suitable grid.
- Floating-point movable grid. Distinguish a fixed-point grid from floating-point's exponent-dependent grid and explain the numerical flexibility versus hardware-cost tradeoff.
- Flexible grids for diverse tensors. Explain why a fixed-point grid cannot represent both small weights and wide-range activations well, and how affine scaling and shifting use the integer range more effectively.
- Affine mapping and parameter derivation. Describe how scale and zero-point relate a stored integer to a real value, and derive them from chosen real and integer endpoints. Explain why integer rounding of the zero-point can nudge the represented range.
- Exact representation of zero. Justify why real zero must map exactly to an integer zero-point, and describe why a calibration range may need to be extended to include zero.
- Zero-point arithmetic overhead. Explain why affine zero-points add terms to an integer dot product and why setting the weight zero-point to zero removes a costly input reduction while allowing asymmetric activations.
- Granular and overload errors. Distinguish rounding error for values inside the quantization grid from clipping error for values outside it, including how each error grows.
- Clipping boundary tradeoff. Explain how widening or tightening the clipping range changes granular and overload error, and why calibration may deliberately clip a small tail.
- Symmetric grids for weights. Explain why symmetric quantization usually fits weights well: their values tend to form a bell-shaped distribution centered near zero, so a zero-centered grid wastes little range.
- Asymmetric grids for activations. Explain why one-sided or mostly positive activations favor an asymmetric grid: a symmetric signed grid spends much of its range on negative values that ReLU outputs do not use.
- Hybrid distribution and runtime tradeoff. Justify the Hybrid Consensus of symmetric weights and asymmetric activations by connecting distribution fit to dot-product cost. Explain that setting the weight zero point to zero removes the runtime sum over quantized inputs, while shifting the activation grid preserves useful range for skewed data.
- Symmetric Weight Quantization. Explain why centered weights suit a symmetric integer grid and describe how the weight quantizer calibrates and quantizes them.
- Affine Activation Grid. Explain why the section uses an affine grid for activations and how its calibration preserves real zero while avoiding wasted codes on ReLU outputs.
- Linear Layer Zero-Point Correction. Trace the quantized linear layer's calculation and explain why its activation zero-point correction matters for shifted inputs.
What it builds on
- Why quantization is the practical response (chapter 1) · The motivation for using quantization to make inference more efficient.
Section by section
Introduction
This chapter introduction frames quantization as the translation from floating-point neural-network values to the small integers used for inference.
- Floating-point to integer translation: Explain that quantization maps continuous floating-point neural-network values onto a finite grid of integer levels for deployment, where inference hardware uses small integer containers.
Why it matters: This distinction gives you the starting point for understanding why quantization requires a mapping.
Key terms: floating-point, quantization, integer levels
2.1 The fixed-point world
This section explains how fixed-point numbers are encoded and computed, why their fixed grid creates range and rounding tradeoffs, and how floating-point uses a movable grid instead.
- Q-format integer interpretation: Explain how a signed Q3.4 value stores an integer while interpreting it with a scale of 1/16, and use that rule to encode or decode a value.
Why it matters: This convention lets you interpret raw integer operations as computations on fractional values. - Two's complement boundaries: Describe how two's complement interprets an eight-bit signed pattern, including why all ones is −1 and what happens when +127 is incremented.
Why it matters: Recognizing the wrap boundaries prevents raw bit patterns from being mistaken for ordinary unsigned values. - Fixed-point multiplication and rounding: Trace why multiplying two Q3.4 raw integers produces an extra-scaled result, how a right shift restores Q3.4, and why adding a rounding bias before the shift can reduce systematic truncation drift.
Why it matters: It explains why correct integer multiplication needs both scale management and a rounding choice. - Fixed-grid range and precision: Explain the range-versus-precision tradeoff when allocating a fixed number of bits, and why measured activation ranges help choose a suitable grid.
Why it matters: Choosing a grid without regard to the values can erase information at either end of the range. - Floating-point movable grid: Distinguish a fixed-point grid from floating-point's exponent-dependent grid and explain the numerical flexibility versus hardware-cost tradeoff.
Why it matters: It shows both why a single fixed grid struggles with widely varying magnitudes and why simpler integer arithmetic is attractive. - FP16 and BF16 tradeoffs: Distinguish FP16's finer mantissa and narrower range from BF16's coarser mantissa and wider range, including the section's stated implication for training and later INT8 quantization.
Why it matters: Equal bit widths do not mean equal numerical behavior or equal sensitivity to later quantization.
Key terms: fixed-point, Q-format, scale factor, Q3.4, two's complement, sign bit, Overflow Cliff, -1 Bridge, wide_product, right shift
2.2 Define the affine mapping with scale and zero point
This section derives affine quantization from scale and zero-point, then explains its accuracy benefits, exact-zero constraint, granularity choices, and arithmetic costs.
- Flexible grids for diverse tensors: Explain why a fixed-point grid cannot represent both small weights and wide-range activations well, and how affine scaling and shifting use the integer range more effectively.
Why it matters: It explains why the extra flexibility of affine quantization is useful in neural networks. - Affine mapping and parameter derivation: Describe how scale and zero-point relate a stored integer to a real value, and derive them from chosen real and integer endpoints. Explain why integer rounding of the zero-point can nudge the represented range.
Why it matters: These relationships let you interpret stored integers and calculate a quantizer for a selected range. - Exact representation of zero: Justify why real zero must map exactly to an integer zero-point, and describe why a calibration range may need to be extended to include zero.
Why it matters: Representing zero exactly protects both convolution boundaries and the benefits of sparse activations. - Quantization parameter granularity: Distinguish per-tensor, per-channel, and group-wise choices for scale and zero-point, including their outlier and hardware trade-offs.
Why it matters: Parameter granularity determines how much an outlier harms other values and how complex execution becomes. - Zero-point arithmetic overhead: Explain why affine zero-points add terms to an integer dot product and why setting the weight zero-point to zero removes a costly input reduction while allowing asymmetric activations.
Why it matters: It shows why a mapping that fits data well can still be more expensive to execute. - Rounding noise and saturation: Distinguish ordinary round-trip quantization error within a calibrated range from clipping error for values outside that range.
Why it matters: It prevents treating a clipped value as though it merely suffered ordinary rounding noise.
Key terms: fixed-point, affine quantization, scale, zero-point, ReLU, dequantization, quantization, scale factor S, zero-point Z, rmin
2.3 Characterize granular and overload errors
This section explains the two kinds of quantization error and how clipping boundaries trade rounding precision against clipping risk.
- Granular and overload errors: Distinguish rounding error for values inside the quantization grid from clipping error for values outside it, including how each error grows.
Why it matters: The two errors call for different choices about the grid’s range and precision. - Clipping boundary tradeoff: Explain how widening or tightening the clipping range changes granular and overload error, and why calibration may deliberately clip a small tail.
Why it matters: A calibration range is a choice about where quantization error will fall, not a way to eliminate it. - Outlier driven quantization risk: Explain why a few large LLM activation outliers can make naive INT8 quantization damaging whether the grid includes or clips them.
Why it matters: Outliers make the ordinary range-versus-precision tradeoff especially severe for transformers. - Histogram based error diagnosis: Describe how to compare an activation histogram with proposed clipping boundaries to anticipate granular and overload risk.
Why it matters: The plot makes the likely cost of a proposed boundary visible before committing to it.
Key terms: granular error, overload error, step size S, clipping, clipping boundary, percentile-based calibration, outliers, LLMs, naive INT8, scale
2.4 Compare symmetric and asymmetric choices
This section explains why the shape of weight and activation distributions leads to a practical choice of symmetric weights and asymmetric activations.
- Symmetric grids for weights: Explain why symmetric quantization usually fits weights well: their values tend to form a bell-shaped distribution centered near zero, so a zero-centered grid wastes little range.
Why it matters: The distribution of the weights explains why a simpler grid need not sacrifice much accuracy. - Asymmetric grids for activations: Explain why one-sided or mostly positive activations favor an asymmetric grid: a symmetric signed grid spends much of its range on negative values that ReLU outputs do not use.
Why it matters: A grid that follows activation values avoids spending codes on a range the tensor does not occupy. - Hybrid distribution and runtime tradeoff: Justify the Hybrid Consensus of symmetric weights and asymmetric activations by connecting distribution fit to dot-product cost. Explain that setting the weight zero point to zero removes the runtime sum over quantized inputs, while shifting the activation grid preserves useful range for skewed data.
Why it matters: This choice balances inference cost against how well each tensor's grid fits its values.
Key terms: weight distributions, bell-shaped distribution, Symmetric Quantization, activations, ReLU, asymmetric quantization, bit-depth, Hybrid Consensus, symmetric weights, asymmetric activations
2.5 End-to-end INT8 inference in PyTorch
The section implements hybrid INT8 quantization in a PyTorch linear layer and uses grid visualizations and FP32 comparisons to explain code allocation, output error, and zero-point correction.
- Symmetric Weight Quantization: Explain why centered weights suit a symmetric integer grid and describe how the weight quantizer calibrates and quantizes them.
Why it matters: This explains both why the weight grid fits the data and why its arithmetic is simpler. - Affine Activation Grid: Explain why the section uses an affine grid for activations and how its calibration preserves real zero while avoiding wasted codes on ReLU outputs.
Why it matters: Matching the grid to the activation distribution gives finer steps without increasing eight-bit storage. - Linear Layer Zero-Point Correction: Trace the quantized linear layer's calculation and explain why its activation zero-point correction matters for shifted inputs.
Why it matters: A layer can appear correct on ReLU inputs yet produce biased answers when a nonzero activation zero-point exposes an omitted term. - FP32 Output Comparison: Describe how the experiment compares the quantized layer with an FP32 baseline and what its error summaries and scatter plot reveal.
Why it matters: The comparison separates the size of quantization error from signs that the outputs are systematically wrong.
Key terms: symmetric quantization, absolute maximum, scale, zero-point, [-127, 127], affine quantizer, asymmetric quantization, zero-point nudge, ReLU Trap, granular error
2.6 Summary
Where this is used later
- Build a calibration set that actually represents production (chapter 4) · Motivates using representative data to observe activation ranges.
- (chapter intro) (chapter 3) · Introduces scale as part of the quantization mapping.
- Quantize activations under outliers and dynamic ranges (chapter 3) · Uses the relationship between scale and representable values.
- Estimate ranges (absolute max, percentile, and MSE) (chapter 4) · Provides a constraint when choosing ranges for quantization.
- Quantize weights with per-tensor and per-channel choices (chapter 3) · Develops per-tensor and per-channel weight choices.
- Identify where precision loss matters (chapter 3) · Assumes the distinction between quantization error within a range and clipping outside it.
- Apply outlier-aware weight paths (chapter 7) · Outlier-aware methods address the risk illustrated here.
- Know where frameworks diverge (chapter 6) · Assumes the distinction between symmetric and asymmetric quantization.
Concepts explained in this chapter
- number roundingRounding maps a numerical value to a nearby representable value according to a specified tie-breaking or directional rule.
- quantization errorGranular error is the bounded quantization error caused by rounding a value to a nearby grid point.
- zero-point nudgeA zero-point nudge is the book's term for the adjustment of an asymmetric quantizer's zero-point to fit a shifted value distribution.
- quantization scaleA scale factor is the multiplicative parameter that maps between quantized or fixed-point integers and their represented numerical values.
- quantization calibrationCalibration is the process of measuring representative model values to determine their actual numerical range for configuring quantization.
- affine quantizationAffine quantization maps real values to discrete integer levels using a scale and a zero-point offset.
- asymmetric quantizationAsymmetric quantization maps values to a grid whose range and zero-point can be shifted to fit a non-centered distribution.
- symmetric quantizationSymmetric quantization uses a zero-centered representation, typically forcing the weight zero-point to zero.
- zero-pointA zero-point is the integer level corresponding to real zero in an affine quantization mapping.
- hybrid consensusHybrid Consensus is a quantization strategy that applies symmetric quantization to weights and asymmetric quantization to activations.
- clippingClipping limits a numerical value to a specified minimum and maximum so it remains within a representable range.
- quantizationQuantization is the representation of numerical values such as neural-network weights and activations with reduced-precision formats.
- exact zero constraintThe exact zero constraint requires real zero to map to an integer value without quantization error.
- fixed-point arithmeticFixed-point arithmetic performs calculations on values encoded with a fixed scaling factor.
- activationA numerical output produced by a neural-network layer or operation and passed to subsequent computation.
- zero-point correctionA zero-point correction is the arithmetic adjustment that compensates for an activation zero-point in quantized matrix multiplication.
- fixed-point representationFixed-point representation encodes numerical values with a fixed scaling position between integer and fractional components.
- floating-point valueA numerical representation that approximates real values using a significand and an exponent.
- precisionThe numerical detail available to represent a value, often determined by the number of bits used.
- q-formatQ-format notation specifies how a fixed-point number's bits are allocated between its integer and fractional components.
- right-shift operationA right shift moves an integer’s bits toward lower-order positions, approximately dividing by a power of two.
- rounding biasRounding bias is a half-step offset added before integer rescaling so that a shift approximates rounding rather than truncation.
- overload errorOverload error is quantization error caused when a value falls outside the representable range and is clipped to a boundary.
- percentile-based calibrationPercentile calibration sets a quantization range using selected distribution percentiles, excluding values beyond those boundaries.
- affine transformationAn affine transformation is a mapping that combines a linear transformation with an additive offset.
- quantization gridA quantization grid is a discrete set of representable numerical levels with a defined spacing.
- block quantizationA technique that assigns scaling parameters separately to fixed-size blocks of numerical values rather than to an entire tensor.
- per-channel quantizationPer-channel quantization assigns separate quantization parameters to each tensor channel.
- per-tensor quantizationPer-tensor quantization uses one set of quantization parameters for all values in a tensor.
- step valueA step size is the spacing between adjacent representable levels in a uniform quantization grid.
- saturationSaturation is the clamping of a value outside a numerical format's representable range to a boundary value.
- bfloat16A 16-bit floating-point format with 8 exponent bits and 7 fraction bits, giving it a dynamic range similar to FP32.
- dynamic rangeDynamic range is the span of magnitudes that a numerical representation can encode.
- exponentThe scale-bearing component of a floating-point representation that determines the magnitude of its significand.
- float16A 16-bit floating-point format with 5 exponent bits and 10 fraction bits.
- integer computationArithmetic performed on integer-valued representations with discrete, fixed-step values.
- mantissaThe part of a floating-point representation that stores the number's significant digits.
- rangeRange is the interval of numerical values that a representation can encode.
- resolutionResolution is the spacing between adjacent representable numerical values.
- two's complementTwo's complement representation encodes signed integers so that the highest-order bit contributes a negative power-of-two weight.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
