liveBook · Manning

Per-token quantization

As explained in Quantization and Fast Inference

Per-token quantization assigns quantization parameters separately to the values associated with each input token.

Per-token quantization gives each token’s activation values their own scale. In transformers, this prevents a high-magnitude token from forcing a coarse scale on quieter tokens. Dynamic per-token quantization recomputes those scales for each input, trading extra computation and one scale per token for better precision. section 3.3

For symmetric INT8 quantization of activations shaped [batch, sequence, hidden], take the absolute maximum across the hidden dimension for each token. Divide each maximum by 127 to obtain scales shaped [batch, sequence, 1]; use 1 for an all-zero token. Divide by the broadcast scale, round, and clamp to [-127, 127]. Dequantize by multiplying each integer by its token’s scale. section 3.3

Listing 3.21 Per-token quantizer
class PerTokenQuantizer:
    def __init__(self, bits=8):
        self.q_max = (1 << (bits - 1)) - 1
    
    def quantize(self, tensor):
        """tensor: [batch, seq_len, hidden_dim]"""
        abs_max = tensor.abs().amax(dim=-1, keepdim=True)         #A
        scales = abs_max / self.q_max
        scales = torch.where(scales == 0, torch.ones_like(scales), scales)
        
        q_tensor = torch.clamp(
            torch.round(tensor / scales), -self.q_max, self.q_max
        ).to(torch.int8)
        
        return q_tensor, scales                                    #B
    
    def dequantize(self, q_tensor, scales):
        return q_tensor.float() * scales
 

Inspect the quantizer’s reduction, zero-scale handling, broadcasting, and dequantization. section 3.3

Different token magnitudes

In a synthetic batch with four amplified token positions, dynamic per-tensor quantization has MSE 0.001629; dynamic per-token quantization has MSE 0.000099, a 16.46× reduction. Normal-token scales average 0.0268, versus 0.1229 for amplified tokens. Both methods share the same maximum scale, but per-token scaling avoids using it for every token. section 3.3

Per-token scaling does not by itself isolate outlier feature dimensions. An outlier-aware mixed-precision path can identify those dimensions in the activations, compute them separately at higher precision, and use per-token activation scales and per-channel weight scales for the normal INT8 path. The two matrix-multiplication outputs are then summed. section 7.2

Use dynamic per-token scaling for transformer activations when its accuracy benefit justifies the scale storage and computation. For KV-cache quantization, values use per-token granularity while keys use per-channel granularity. Verify hardware compatibility before choosing a deployment configuration. section 3.3 section 3.7

How Quantization and Fast Inference covers it

How deep it goes
Implements symmetric absmax quantization and dequantization, compares reconstruction MSE, and incorporates row-wise scaling into a mixed-precision matrix multiplication.
Examples
Compares a synthetic batch with amplified tokens and tests an outlier-aware INT8 path on OPT-6.7B activations.
What you can do after
Builds a symmetric per-token quantizer from scratch and evaluates its reconstruction error against per-tensor alternatives.
How it connects
Connects activation quantization to per-tensor quantization, per-channel weight quantization, outlier handling, and KV-cache quantization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.