A quantization grid is the set of numerical levels available to represent values. A uniform integer grid uses fixed spacing across a chosen range: values must round to a grid point or be clipped at a boundary. Choosing the grid trades range against resolution. section 1.3.2 section 2.3
Unlike a uniform integer grid, floating-point spacing is denser near zero and sparser at larger magnitudes. Fixed-width integer arithmetic can use simpler, denser hardware and typically lower energy per operation, but the resulting accuracy depends on how well the grid fits the data. section 1.3.2 section 1.3
Two errors determine the placement tradeoff:
- Granular error occurs when an in-range value rounds to a grid point. With nearest rounding and uniform step size S, its magnitude is at most S/2. section 2.3
- Overload error occurs when an out-of-range value is clipped to the boundary. Its magnitude grows with distance beyond that boundary. section 2.3
- With a fixed number of levels, widening the range reduces clipping but coarsens spacing; tightening it improves resolution inside the range but clips more values. Choose the boundary for the data distribution. section 2.3
Rounding and clipping
With S = 0.05, an in-range value of −0.12 rounds to −0.10, an error of −0.02, within the 0.025 bound. By contrast, an attention logit of 60 clipped to a range ending at 20 becomes 20; that distortion can substantially flatten the subsequent softmax. section 2.3
Grid placement also depends on the tensor. A symmetric INT8 weight grid can use codes [−127, 127], zero-point 0, and a scale based on the absolute maximum. For nonnegative ReLU activations, an asymmetric grid can use [0, 255] instead of leaving negative codes unused; in one comparison its step was 0.069 rather than 0.139, approximately twice the resolution. section 2.5
At low bit widths, scaling granularity matters as well as spacing. In one weight comparison, INT4 per-tensor scaling had mean squared error 2.80×10⁻⁵, versus 4.56×10⁻⁶ for INT4 group-128. An outlier can set a single tensor-wide scale and leave many levels poorly used. Compare formats at the same scaling granularity before attributing gains to spacing alone. section 8.3.3
Quantization-aware training adapts weights to the grid rather than only accepting post-training rounding. Fake quantization simulates the quantize-dequantize step during the forward pass; observers track ranges used to calculate scale and zero-point. In an INT4 CNN example, weights migrated toward grid points during training, reducing rounding error. section 5.2.1 section 5.0
Figure 2.5 The granular-versus-overload error tradeoff. Widening the quantization grid (increasing the clipping range) reduces overload error—fewer values are clipped—but coarsens the step size, increasing granular error for values inside the range. Tightening the grid does the reverse. The optimal clipping boundary minimizes total distortion and depends on the shape of the data distribution: heavy-tailed distributions push the boundary outward; tightly concentrated distributions pull it inward.

The figure shows how changing the clipping range shifts granular and overload error. section 2.3
Figure 5.9 Weight distributions migrate toward INT4 grid points during QAT. Five snapshots of conv1 weights (epochs 0–49) show the initially Gaussian distribution reorganizing into sharp spikes aligned with quantization levels (dashed lines), demonstrating that QAT actively adapts weights to minimize rounding error rather than passively tolerating it.

The snapshots show weights aligning with INT4 grid levels during training. section 5.2.1
