Per-token quantization gives each token’s activation values their own scale. In transformers, this prevents a high-magnitude token from forcing a coarse scale on quieter tokens. Dynamic per-token quantization recomputes those scales for each input, trading extra computation and one scale per token for better precision. section 3.3
For symmetric INT8 quantization of activations shaped [batch, sequence, hidden], take the absolute maximum across the hidden dimension for each token. Divide each maximum by 127 to obtain scales shaped [batch, sequence, 1]; use 1 for an all-zero token. Divide by the broadcast scale, round, and clamp to [-127, 127]. Dequantize by multiplying each integer by its token’s scale. section 3.3
Listing 3.21 Per-token quantizer
class PerTokenQuantizer:
def __init__(self, bits=8):
self.q_max = (1 << (bits - 1)) - 1
def quantize(self, tensor):
"""tensor: [batch, seq_len, hidden_dim]"""
abs_max = tensor.abs().amax(dim=-1, keepdim=True) #A
scales = abs_max / self.q_max
scales = torch.where(scales == 0, torch.ones_like(scales), scales)
q_tensor = torch.clamp(
torch.round(tensor / scales), -self.q_max, self.q_max
).to(torch.int8)
return q_tensor, scales #B
def dequantize(self, q_tensor, scales):
return q_tensor.float() * scales
Inspect the quantizer’s reduction, zero-scale handling, broadcasting, and dequantization. section 3.3
Different token magnitudes
In a synthetic batch with four amplified token positions, dynamic per-tensor quantization has MSE 0.001629; dynamic per-token quantization has MSE 0.000099, a 16.46× reduction. Normal-token scales average 0.0268, versus 0.1229 for amplified tokens. Both methods share the same maximum scale, but per-token scaling avoids using it for every token. section 3.3
Per-token scaling does not by itself isolate outlier feature dimensions. An outlier-aware mixed-precision path can identify those dimensions in the activations, compute them separately at higher precision, and use per-token activation scales and per-channel weight scales for the normal INT8 path. The two matrix-multiplication outputs are then summed. section 7.2
Use dynamic per-token scaling for transformer activations when its accuracy benefit justifies the scale storage and computation. For KV-cache quantization, values use per-token granularity while keys use per-channel granularity. Verify hardware compatibility before choosing a deployment configuration. section 3.3 section 3.7
