liveBook · Manning

Entropy calibration

As explained in Quantization and Fast Inference

Entropy calibration selects a quantization range by optimizing an entropy-based proxy for the distortion between original and quantized distributions.

Entropy calibration chooses a quantization range by minimizing KL-divergence, preserving distribution shape rather than minimizing point-wise reconstruction error. That objective can fail on heavy-tailed activations: a tall peak near zero may lead the optimizer to shrink the range and clip sparse tails. section 4.3

Heavy-tailed activation failure

For the reported BERT activation distribution, entropy calibration chose a range of 0.7199 and clipped about 25% of values. Its MSE was 0.160087, versus 0.000600 for MSE-optimal calibration. section 4.3

CALIBRATION RESULTS
──────────────────────────────────────────────────────────────
Method          Range      Scale        Clipped    MSE          SNR (dB)  
──────────────────────────────────────────────────────────────
mse             10.5917    0.083399     0.0236%    0.000600     29.52     
percentile      10.7027    0.084273     0.0101%    0.000604     29.49     
absmax          11.5178    0.090691     0.0000%    0.000688     28.93     
entropy         0.7199     0.005668     24.9989%   0.160087     5.26      
──────────────────────────────────────────────────────────────
Best method: mse (MSE: 0.000600)

Compare the ranges, clipped fractions, and MSE values for entropy, MSE, percentile, and absmax calibration. section 4.3

Entropy calibration can work for roughly uniform or Gaussian distributions, but requires caution on heavy-tailed transformer activations. Check the observed distribution and validate clipping and reconstruction error before choosing it. section 4.3

How Quantization and Fast Inference covers it

How deep it goes
Explains the KL objective and a measured heavy-tail failure, but does not detail entropy observer implementation.
Examples
Compares entropy and MSE calibration on BERT activations, including range, clipping, and reconstruction-error measurements.
What you can do after
Evaluate entropy calibration against MSE by checking clipped fractions and reconstruction error for the target distribution.
How it connects
Connects quantization calibration to KL-divergence and MSE-based range selection.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.