liveBook · Manning

Absmax quantization

As explained in Quantization and Fast Inference

A quantization range-estimation method that sets boundaries using the largest observed absolute value.

Absmax calibration sets a quantization range from the largest observed absolute value. It is simple and avoids clipping calibration values, but a single outlier can make the quantization steps too coarse for most values. section 4.3

For symmetric INT8 quantization, compute A = max(|x|), then set the range to [-A, A] and the scale to S = A/127. Reconstruct an observed value with x̂ = S · round(x/S). Calibration values have no clipping error, and their rounding error is bounded by S/2. Production values can still exceed the calibrated range and be clipped. section 4.3

An outlier stretches the scale

For BERT-base layer 6 FFN activations, the absolute maximum was 11.52 and the 99th percentile was 1.92: an outlier ratio of 5.99. Absmax gives an INT8 scale of approximately 0.0907 and a rounding-error bound of approximately 0.0454. In the measured sweep, absmax produced MSE 0.000688 with no clipping; a 99.99th-percentile range produced MSE 0.000604 while clipping approximately 0.0101% of values. section 4.3

Listing 4.4 Analyzing absmax limitations
def analyze_absmax_problem(model, tokenizer, texts):
    activations = []
    
    def hook(module, input, output):
        tensor = output[0] if isinstance(output, tuple) else output
        activations.append(tensor.detach())
    
    handle = model.encoder.layer[6].output.register_forward_hook(hook)  #A
    
    with torch.no_grad():
        for text in texts:
            inputs = tokenizer(text, return_tensors='pt', truncation=True)
            model(**inputs)
    
    handle.remove()
    all_acts = torch.cat([a.flatten() for a in activations])
    
    absmax = all_acts.abs().max().item()
    p99 = torch.quantile(all_acts.abs(), 0.99).item()
    
    return {'absmax': absmax, 'p99': p99, 'outlier_ratio': absmax / p99}  #B

The diagnostic captures BERT layer activations and compares their absolute maximum with the 99th percentile. section 4.3

Use the maximum-to-99th-percentile ratio as a starting diagnostic: below 2×, absmax is generally sufficient; from 2–10×, try 99.99th-percentile clipping; above 10×, try MSE-optimal calibration. Measure the result on representative calibration data and validate accuracy against an FP32 baseline. INT8 storage alone does not improve latency without a suitable execution backend. section 4.3 section 4.5

Absmax can also set a separate weight scale for each channel during quantization-aware training. In a SmallCNN run, the 32 conv1 scales remained stable across 15 epochs; pretrained models may drift early, making observer freezing a useful safeguard. section 5.4.2

How Quantization and Fast Inference covers it

How deep it goes
Derives symmetric INT8 scaling and rounding bounds, then compares outlier sensitivity and measured calibration error against alternative ranges.
Examples
Measures BERT-base layer 6 activations and tracks SmallCNN per-channel scales across 15 training epochs.
What you can do after
Build it from scratch, diagnose outlier sensitivity, and compare its calibration error with clipping alternatives.
How it connects
Connects symmetric quantization with percentile clipping, MSE-optimal calibration, outlier ratios, and per-channel quantization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.