Static quantization fixes activation ranges before deployment: observe representative calibration data, compute scales, then reuse those parameters at runtime. This avoids runtime range-computation overhead, but inputs that drift from calibration can lose accuracy sharply. section 3.3
Use calibration results to manage two practical risks:
- Outliers can consume the available range: for one BERT layer, scaling to the maximum leaves typical activations about 9 quantization levels, while scaling to the 99th percentile gives about 51 but clips outliers. section 3.3
- Distribution shift can make fixed scales unsuitable: a comparison reports dynamic quantization up to 1,126× lower MSE on wide inputs than static quantization. section 3.3
- For transformer deployment, verify which operations are quantized and which precision actually runs; restricting ONNX Runtime quantization to MatMul avoids the cited static default's embedding-lookup issue. section 9.6
Listing 3.18 Activation observer for static quantization
class ActivationObserver:
def __init__(self):
self.min_val = float('inf')
self.max_val = float('-inf')
def observe(self, tensor: torch.Tensor):
self.min_val = min(self.min_val, tensor.min().item())
self.max_val = max(self.max_val, tensor.max().item())
def get_scale_zeropoint(self, bits=8, symmetric=False):
if symmetric:
abs_max = max(abs(self.min_val), abs(self.max_val))
q_max = (1 << (bits - 1)) - 1 #A
scale = abs_max / q_max if abs_max > 0 else 1.0
zero_point = 0
else:
q_min, q_max = 0, (1 << bits) - 1 #B
scale = (self.max_val - self.min_val) / (q_max - q_min)
zero_point = int(round(q_min - self.min_val / scale))
return scale, zero_point
An activation observer illustrates collecting minimum and maximum values and deriving scale and zero point before deployment. section 3.3

