Percentile-based calibration chooses quantization boundaries from a distribution percentile rather than its observed extremes. It sacrifices some tail values to give the bulk of activations a finer quantization grid. The gain in rounding precision must outweigh the error from clipping. section 4.3 section 2.3
For symmetric INT8 calibration, collect representative activations, take their absolute values, and select a percentile threshold T. Use the range [−T, T] and scale S = T/127. Values beyond the range are clipped; values inside it are rounded to the grid. A 99.9th-percentile threshold clips approximately 0.1% of the calibration values by magnitude. section 4.3
Listing 4.5 Percentile-based observer
class PercentileObserver:
def __init__(self, percentile=99.99):
self.percentile = percentile
self.all_values = []
def observe(self, tensor):
self.all_values.append(tensor.abs().flatten().cpu())
def compute_range(self):
all_abs = torch.cat(self.all_values)
return torch.quantile(all_abs, self.percentile / 100).item() #AThis PyTorch observer collects absolute activation values and computes the selected percentile threshold. section 4.3
A percentile sweep
For BERT layer 6 activations, the absolute maximum is 11.5178 and the 99th-percentile magnitude is 1.9227. Using the maximum gives scale 0.090691 and MSE 0.000688. The 99.99th percentile gives range 10.7027, scale 0.084273, and MSE 0.000604 while clipping 0.0101% of values. The more aggressive 99th percentile clips 1.0001% and increases MSE to 0.068384. The tightest range is not necessarily the best. section 4.3
Choose and validate the boundary against the observed distribution:
- Compute the outlier ratio as maximum absolute activation divided by its 99th-percentile magnitude. Below 2×, absolute maximum may suffice; from 2× to 10×, 99.99% is a conservative starting point; above 10×, consider MSE-optimal calibration. section 4.3
- Sweep percentiles and compare clipping, reconstruction MSE, and model behavior rather than assuming that fewer clipped values or a smaller scale is always better. section 4.3 section 2.3
- Watch for distribution drift and recalibrate when needed. Production activations can exceed a boundary estimated from calibration data. section 2.6 section 4.3
Clipping is especially risky when rare outlier channels carry important information. In OPT-6.7B, large coordinated channels recur across layers; clipping them can collapse model behavior. Per-channel scaling or outlier-aware methods address a different problem than simply tightening a per-tensor boundary. section 7.1 section 2.3

