liveBook · Manning

Percentile-based calibration

As explained in Quantization and Fast Inference

Percentile calibration sets a quantization range using selected distribution percentiles, excluding values beyond those boundaries.

Percentile-based calibration chooses quantization boundaries from a distribution percentile rather than its observed extremes. It sacrifices some tail values to give the bulk of activations a finer quantization grid. The gain in rounding precision must outweigh the error from clipping. section 4.3 section 2.3

For symmetric INT8 calibration, collect representative activations, take their absolute values, and select a percentile threshold T. Use the range [−T, T] and scale S = T/127. Values beyond the range are clipped; values inside it are rounded to the grid. A 99.9th-percentile threshold clips approximately 0.1% of the calibration values by magnitude. section 4.3

Listing 4.5 Percentile-based observer
class PercentileObserver:
    def __init__(self, percentile=99.99):
        self.percentile = percentile
        self.all_values = []
    
    def observe(self, tensor):
        self.all_values.append(tensor.abs().flatten().cpu())
    
    def compute_range(self):
        all_abs = torch.cat(self.all_values)
        return torch.quantile(all_abs, self.percentile / 100).item()  #A

This PyTorch observer collects absolute activation values and computes the selected percentile threshold. section 4.3

A percentile sweep

For BERT layer 6 activations, the absolute maximum is 11.5178 and the 99th-percentile magnitude is 1.9227. Using the maximum gives scale 0.090691 and MSE 0.000688. The 99.99th percentile gives range 10.7027, scale 0.084273, and MSE 0.000604 while clipping 0.0101% of values. The more aggressive 99th percentile clips 1.0001% and increases MSE to 0.068384. The tightest range is not necessarily the best. section 4.3

Choose and validate the boundary against the observed distribution:

  • Compute the outlier ratio as maximum absolute activation divided by its 99th-percentile magnitude. Below 2×, absolute maximum may suffice; from 2× to 10×, 99.99% is a conservative starting point; above 10×, consider MSE-optimal calibration. section 4.3
  • Sweep percentiles and compare clipping, reconstruction MSE, and model behavior rather than assuming that fewer clipped values or a smaller scale is always better. section 4.3 section 2.3
  • Watch for distribution drift and recalibrate when needed. Production activations can exceed a boundary estimated from calibration data. section 2.6 section 4.3

Clipping is especially risky when rare outlier channels carry important information. In OPT-6.7B, large coordinated channels recur across layers; clipping them can collapse model behavior. Per-channel scaling or outlier-aware methods address a different problem than simply tightening a per-tensor boundary. section 7.1 section 2.3

How Quantization and Fast Inference covers it

How deep it goes
Explains threshold computation and error tradeoffs, then compares percentile sweeps with alternative range estimators.
Examples
Uses BERT layer 6 activation statistics and a percentile sweep to compare clipping, scale, and reconstruction error.
What you can do after
Apply it via a PyTorch percentile observer and evaluate thresholds using clipping rates and reconstruction error.
How it connects
Connects percentile clipping to symmetric quantization, mean-squared-error calibration, entropy calibration, and outlier-aware quantization.

Also explained in

Compare how 2 books explain percentile-based calibration →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.