Absmax calibration sets a quantization range from the largest observed absolute value. It is simple and avoids clipping calibration values, but a single outlier can make the quantization steps too coarse for most values. section 4.3
For symmetric INT8 quantization, compute A = max(|x|), then set the range to [-A, A] and the scale to S = A/127. Reconstruct an observed value with x̂ = S · round(x/S). Calibration values have no clipping error, and their rounding error is bounded by S/2. Production values can still exceed the calibrated range and be clipped. section 4.3
An outlier stretches the scale
For BERT-base layer 6 FFN activations, the absolute maximum was 11.52 and the 99th percentile was 1.92: an outlier ratio of 5.99. Absmax gives an INT8 scale of approximately 0.0907 and a rounding-error bound of approximately 0.0454. In the measured sweep, absmax produced MSE 0.000688 with no clipping; a 99.99th-percentile range produced MSE 0.000604 while clipping approximately 0.0101% of values. section 4.3
Listing 4.4 Analyzing absmax limitations
def analyze_absmax_problem(model, tokenizer, texts):
activations = []
def hook(module, input, output):
tensor = output[0] if isinstance(output, tuple) else output
activations.append(tensor.detach())
handle = model.encoder.layer[6].output.register_forward_hook(hook) #A
with torch.no_grad():
for text in texts:
inputs = tokenizer(text, return_tensors='pt', truncation=True)
model(**inputs)
handle.remove()
all_acts = torch.cat([a.flatten() for a in activations])
absmax = all_acts.abs().max().item()
p99 = torch.quantile(all_acts.abs(), 0.99).item()
return {'absmax': absmax, 'p99': p99, 'outlier_ratio': absmax / p99} #BThe diagnostic captures BERT layer activations and compares their absolute maximum with the 99th percentile. section 4.3
Use the maximum-to-99th-percentile ratio as a starting diagnostic: below 2×, absmax is generally sufficient; from 2–10×, try 99.99th-percentile clipping; above 10×, try MSE-optimal calibration. Measure the result on representative calibration data and validate accuracy against an FP32 baseline. INT8 storage alone does not improve latency without a suitable execution backend. section 4.3 section 4.5
Absmax can also set a separate weight scale for each channel during quantization-aware training. In a SmallCNN run, the 32 conv1 scales remained stable across 15 epochs; pretrained models may drift early, making observer freezing a useful safeguard. section 5.4.2
