liveBook · Manning

Overload error

As explained in Quantization and Fast Inference

Overload error is quantization error caused when a value falls outside the representable range and is clipped to a boundary.

Overload error occurs when a value falls outside a quantization grid and is clipped to its boundary. Unlike rounding error inside the grid, clipping error has no upper bound: the farther a value lies beyond the boundary, the larger the distortion. section 2.3

Clipping can introduce structured distortion and nonlinearities that propagate through layers. Choosing a wider range clips fewer values but coarsens the grid; tightening it improves precision inside the range but clips more values. section 2.3 section 2.6

Clipped attention logits

If attention logits spike to 60 while the calibrated range is [−20, 20], each spike is clipped to 20. The resulting softmax weights can become much flatter, disrupting attention to the intended token. section 2.3

Figure 2.5 The granular-versus-overload error tradeoff. Widening the quantization grid (increasing the clipping range) reduces overload error—fewer values are clipped—but coarsens the step size, increasing granular error for values inside the range. Tightening the grid does the reverse. The optimal clipping boundary minimizes total distortion and depends on the shape of the data distribution: heavy-tailed distributions push the boundary outward; tightly concentrated distributions pull it inward.

Compare how widening or tightening the clipping range shifts distortion between overload and granular error. section 2.3

How Quantization and Fast Inference covers it

How deep it goes
Explains clipping, unbounded error, downstream effects, and its tradeoff with granular error.
Examples
Uses attention logits spiking to 60 against a calibrated range of [−20, 20].
What you can do after
Diagnose clipping exposure by comparing activation histograms with candidate quantization boundaries.
How it connects
Connects quantization range selection to granular error, calibration, and softmax behavior.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.