Quantization represents weights and activations with fewer bits, often replacing 16-bit or 32-bit floating point with 8-bit or 4-bit integers. For memory-bound inference, smaller representations reduce the data stored and moved per token. A 16-bit-to-8-bit weight conversion halves raw weight storage; actual latency and energy gains depend on the hardware and runtime. section 1.0 section 1.2 section 12.4
Integer quantization maps values onto a finite, uniformly spaced grid. An affine mapping uses a scale S for grid spacing and a zero point z for the integer representing real zero: q = clip(round(x/S) + z), with reconstruction x̂ = S(q − z). Values between grid points round; values outside the supported range clip. Symmetric and asymmetric schemes make different choices about this mapping. section 1.3 section 2.0 section 2.3
Fixed-point rounding
In Q3.4, the grid spacing is 1/16 = 0.0625. Encoding 0.3 gives round(0.3 × 16) = 5, which reconstructs as 5/16 = 0.3125: an error of +0.0125. Multiplying two encoded values produces a wider integer that must be rescaled; truncating during that step can introduce systematic downward error. section 2.1
For nearest rounding inside the grid, granular error has magnitude at most S/2. Overload error occurs when values are clipped outside the grid and can grow with their distance from the boundary. At a fixed bit-width, widening the range reduces clipping but makes the grid coarser; narrowing it improves resolution but risks losing outliers. section 2.3 section 1.3.2
Figure 2.5 The granular-versus-overload error tradeoff. Widening the quantization grid (increasing the clipping range) reduces overload error—fewer values are clipped—but coarsens the step size, increasing granular error for values inside the range. Tightening the grid does the reverse. The optimal clipping boundary minimizes total distortion and depends on the shape of the data distribution: heavy-tailed distributions push the boundary outward; tightly concentrated distributions pull it inward.

The diagram shows how the clipping boundary trades granular error against overload error. section 2.3
Choose precision and granularity separately for each target. Weights are static at inference, activations vary by input and may contain outliers, and the transformer KV cache is written per token and repeatedly read. A per-tensor scale is simple; per-channel and group-wise scales can preserve more precision, with choices constrained by tensor statistics, accuracy requirements, and hardware. section 3.0
For large language models, LLM.int8() separates outlier dimensions into an FP16 path; GPTQ redistributes weight-rounding error; AWQ rescales channels associated with high-magnitude activations. Validate the chosen format on the target runtime: compare throughput, memory, and quality, and check that execution does not fall back to full precision. section 7.7 section 12.3 section 12.4





















