Activation quantization represents intermediate neural-network tensors at reduced precision, potentially accelerating computation. Unlike frozen weights, activations change with each input, so a range that works for one input may clip another or waste numerical resolution. section 3.1 section 3.3
For symmetric signed 8-bit quantization, a typical scale is the maximum absolute activation divided by 127. Divide each activation by the scale, round and clamp it to −127 through 127; multiply the integer by the scale to approximate the original value. A larger scale covers more extreme values but gives coarser resolution. section 3.3
The outlier tradeoff
In one BERT layer, the median absolute activation is 1.93 and the maximum is 25.81. Scaling by the maximum gives a step of 0.2032, so the median occupies about 9 integer levels. Scaling by the 99th percentile gives a step of 0.0378 and about 51 levels for the median, but clips the largest values. section 3.3
Choose how and when to calculate scales:
- Static per-tensor: observe representative calibration inputs, calculate a scale, and freeze it before deployment. Scale calculation adds no runtime overhead, but distribution shifts can cause clipping or poor resolution. section 3.3
- Dynamic per-tensor: calculate a scale from the current tensor. This adapts to input ranges at the cost of runtime computation; a single outlier can still set a coarse scale for the whole tensor. section 3.3
- Dynamic per-token: calculate one scale across each token’s hidden dimension. Tokens with different ranges no longer share one scale, but the method stores a scale per token. section 3.3
- Static percentile scaling trades clipping of extreme values for finer resolution across typical values. Mixed precision is another option for severe outliers, with higher compute cost. section 3.3
Per-token scaling in a synthetic workload
For hidden states with a few amplified tokens, measured mean squared error was 0.001629 with dynamic per-tensor scaling and 0.000099 with dynamic per-token scaling: a 16.46× reduction in that example. Results depend on the activation distribution. section 3.3
Listing 3.21 Per-token quantizer
class PerTokenQuantizer:
def __init__(self, bits=8):
self.q_max = (1 << (bits - 1)) - 1
def quantize(self, tensor):
"""tensor: [batch, seq_len, hidden_dim]"""
abs_max = tensor.abs().amax(dim=-1, keepdim=True) #A
scales = abs_max / self.q_max
scales = torch.where(scales == 0, torch.ones_like(scales), scales)
q_tensor = torch.clamp(
torch.round(tensor / scales), -self.q_max, self.q_max
).to(torch.int8)
return q_tensor, scales #B
def dequantize(self, q_tensor, scales):
return q_tensor.float() * scales
The per-token implementation computes scales over the hidden dimension and uses them to quantize and reconstruct each token. section 3.3
Measure activation ranges and outliers on representative inputs before choosing a strategy. Static quantization is supported by runtimes including TensorRT, ONNX Runtime, and TFLite; verify the target runtime and hardware before assuming a precision change will improve throughput. section 3.3
