liveBook · Manning

Activation quantization

As explained in Quantization and Fast Inference

Representing neural-network activations with reduced-precision numerical formats.

Activation quantization represents intermediate neural-network tensors at reduced precision, potentially accelerating computation. Unlike frozen weights, activations change with each input, so a range that works for one input may clip another or waste numerical resolution. section 3.1 section 3.3

For symmetric signed 8-bit quantization, a typical scale is the maximum absolute activation divided by 127. Divide each activation by the scale, round and clamp it to −127 through 127; multiply the integer by the scale to approximate the original value. A larger scale covers more extreme values but gives coarser resolution. section 3.3

The outlier tradeoff

In one BERT layer, the median absolute activation is 1.93 and the maximum is 25.81. Scaling by the maximum gives a step of 0.2032, so the median occupies about 9 integer levels. Scaling by the 99th percentile gives a step of 0.0378 and about 51 levels for the median, but clips the largest values. section 3.3

Choose how and when to calculate scales:

  • Static per-tensor: observe representative calibration inputs, calculate a scale, and freeze it before deployment. Scale calculation adds no runtime overhead, but distribution shifts can cause clipping or poor resolution. section 3.3
  • Dynamic per-tensor: calculate a scale from the current tensor. This adapts to input ranges at the cost of runtime computation; a single outlier can still set a coarse scale for the whole tensor. section 3.3
  • Dynamic per-token: calculate one scale across each token’s hidden dimension. Tokens with different ranges no longer share one scale, but the method stores a scale per token. section 3.3
  • Static percentile scaling trades clipping of extreme values for finer resolution across typical values. Mixed precision is another option for severe outliers, with higher compute cost. section 3.3

Per-token scaling in a synthetic workload

For hidden states with a few amplified tokens, measured mean squared error was 0.001629 with dynamic per-tensor scaling and 0.000099 with dynamic per-token scaling: a 16.46× reduction in that example. Results depend on the activation distribution. section 3.3

Listing 3.21 Per-token quantizer
class PerTokenQuantizer:
    def __init__(self, bits=8):
        self.q_max = (1 << (bits - 1)) - 1
    
    def quantize(self, tensor):
        """tensor: [batch, seq_len, hidden_dim]"""
        abs_max = tensor.abs().amax(dim=-1, keepdim=True)         #A
        scales = abs_max / self.q_max
        scales = torch.where(scales == 0, torch.ones_like(scales), scales)
        
        q_tensor = torch.clamp(
            torch.round(tensor / scales), -self.q_max, self.q_max
        ).to(torch.int8)
        
        return q_tensor, scales                                    #B
    
    def dequantize(self, q_tensor, scales):
        return q_tensor.float() * scales
 

The per-token implementation computes scales over the hidden dimension and uses them to quantize and reconstruct each token. section 3.3

Measure activation ranges and outliers on representative inputs before choosing a strategy. Static quantization is supported by runtimes including TensorRT, ONNX Runtime, and TFLite; verify the target runtime and hardware before assuming a precision change will improve throughput. section 3.3

How Quantization and Fast Inference covers it

How deep it goes
Works through scale calculation, calibration, clipping, dynamic scaling, and per-token quantization with numerical error comparisons.
Examples
Uses BERT activation percentiles and synthetic hidden states to compare outlier clipping and per-token scaling.
What you can do after
Evaluate activation quantization strategies using observed ranges, clipping risk, mean squared error, and scale overhead.
How it connects
Connects activation quantization with calibration, outliers, mixed precision, and weight quantization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.