Narrowing a quantization range improves spacing-based precision for in-range values but clips outliers. Percentile clipping trades bounded error on a subset of values for lower rounding error across the majority; the best threshold depends on the distribution and downstream task. section 4.3
BERT-base layer 6 activation sweep
The 99th-percentile threshold clipped about 1% of values and produced MSE 0.068384. At 99.99%, clipping fell to 0.0101% and MSE to 0.000604; absmax clipped none but had higher MSE, 0.000688. The empirical optimum was non-monotonic. section 4.3
PERCENTILE SWEEP ANALYSIS ────────────────────────────────────────────────────────────── Percentile Range Clipped MSE SNR (dB) ────────────────────────────────────────────────────────────── 99.00 1.9227 1.0001% 0.068384 8.95 99.50 2.2657 0.5006% 0.060052 9.52 99.90 6.2455 0.1010% 0.011418 16.73 99.95 9.7443 0.0505% 0.000833 28.09 99.99 10.7027 0.0101% 0.000604 29.49 ← BEST 100.00 11.5178 0.0000% 0.000688 28.93 ────────────────────────────────────────────────────────────── Optimal percentile: 99.99%
Compare clipping, MSE, and signal-to-noise ratio across percentile thresholds. section 4.3
