liveBook · Manning

Clipping-resolution trade-off

As explained in Quantization and Fast Inference

The clipping-resolution trade-off describes how narrowing a quantization range increases clipping while improving the spacing-based precision of values that remain in range.

Narrowing a quantization range improves spacing-based precision for in-range values but clips outliers. Percentile clipping trades bounded error on a subset of values for lower rounding error across the majority; the best threshold depends on the distribution and downstream task. section 4.3

BERT-base layer 6 activation sweep

The 99th-percentile threshold clipped about 1% of values and produced MSE 0.068384. At 99.99%, clipping fell to 0.0101% and MSE to 0.000604; absmax clipped none but had higher MSE, 0.000688. The empirical optimum was non-monotonic. section 4.3

PERCENTILE SWEEP ANALYSIS
──────────────────────────────────────────────────────────────
Percentile   Range      Clipped    MSE          SNR (dB)  
──────────────────────────────────────────────────────────────
99.00        1.9227     1.0001%    0.068384     8.95      
99.50        2.2657     0.5006%    0.060052     9.52      
99.90        6.2455     0.1010%    0.011418     16.73     
99.95        9.7443     0.0505%    0.000833     28.09     
99.99        10.7027    0.0101%    0.000604     29.49      ← BEST
100.00       11.5178    0.0000%    0.000688     28.93     
──────────────────────────────────────────────────────────────
Optimal percentile: 99.99%

Compare clipping, MSE, and signal-to-noise ratio across percentile thresholds. section 4.3

How Quantization and Fast Inference covers it

How deep it goes
Explains rounding and overload error, then shows that total error varies non-monotonically and requires empirical threshold selection.
Examples
Compares percentile thresholds using clipping rates and MSE for BERT-base layer 6 activations.
What you can do after
Apply the trade-off by sweeping range thresholds and measuring clipping and reconstruction error to find an empirical optimum.
How it connects
Connects percentile clipping with quantization, calibration, rounding error, clipping error, and mean squared error.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.