liveBook · Manning

Quantization error

As explained in Quantization and Fast Inference

Granular error is the bounded quantization error caused by rounding a value to a nearby grid point.

Quantization maps values to a finite grid, trading numerical accuracy for smaller representations. Granular error is the rounding error for a value inside that grid; overload error occurs when a value outside the grid is clipped. Calibration must balance both when choosing the range. section 2.3 section 2.0

For uniform nearest-grid-point rounding with step size S, the magnitude of granular error is at most S/2. This bound does not apply to clipped values: overload error grows with the distance beyond the grid boundary. section 2.3

Rounding on a 0.05 grid

With S = 0.05, quantize x by rounding x/S and multiplying by S. For x = −0.12, the result is −0.10, so x minus the quantized value is −0.02. For x = 0.11, the result is 0.10 and the error is 0.01. Both magnitudes are below S/2 = 0.025. section 2.3

With a fixed number of grid levels, widening the range protects more values from clipping but increases the step size and rounding error. Tightening it improves resolution inside the range but clips more of the tails. The best boundary depends on the distribution, not just its maximum. section 2.3

Figure 2.5 The granular-versus-overload error tradeoff. Widening the quantization grid (increasing the clipping range) reduces overload error—fewer values are clipped—but coarsens the step size, increasing granular error for values inside the range. Tightening the grid does the reverse. The optimal clipping boundary minimizes total distortion and depends on the shape of the data distribution: heavy-tailed distributions push the boundary outward; tightly concentrated distributions pull it inward.

The range diagram shows how granular and overload errors compete during calibration. section 2.3

Grid placement matters too. Symmetric quantization can waste codes on the unused negative side of one-sided activations; asymmetric quantization shifts the grid to use those codes, at the cost of zero-point correction. In one ReLU activation comparison, the asymmetric scale was 0.069 versus 0.139 for the symmetric grid, a 2.01× resolution gain. section 2.5 section 2.6

Inspect error by channel when outliers determine the scale. For OPT-6.7B, per-channel INT8 reduced total mean squared error by 1,872× in layer 0 and 442× in layer 31 compared with per-tensor INT8. Outlier dimensions still contributed 82% of residual error in early layers, so reducing the shared-scale penalty did not eliminate the need to protect them. section 7.1

When calibration alone is insufficient, quantization-aware training exposes the model to fake quantization during training. Weights can move toward grid points, reducing rounding error before deployment; retraining is a cost to weigh against simpler interventions. section 5.2.1 section 5.0

How Quantization and Fast Inference covers it

How deep it goes
Works through the nearest-grid error bound, clipping tradeoff, and tensor-level error measurements.
Examples
Uses a 0.05-step rounding example and an OPT-6.7B per-channel error decomposition.
What you can do after
Diagnose quantization error by comparing clipping, grid resolution, and channel-level distortion.
How it connects
Connects rounding error to overload error, symmetric and asymmetric quantization, outliers, and quantization-aware training.

Also explained in

Compare how 3 books explain quantization error →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.