liveBook · Manning

Per-tensor quantization

As explained in Quantization and Fast Inference

Per-tensor quantization uses one set of quantization parameters for all values in a tensor.

Per-tensor quantization uses one set of quantization parameters for every value in a tensor. It minimizes scale metadata and provides a simple baseline, but an outlier can set a coarse resolution for all other values. The tradeoff depends on the tensor’s distribution, precision, and target hardware. section 3.0 section 2.2

In affine quantization, the entire tensor shares a scale S and zero point Z. For symmetric INT8 weight quantization, Z is zero and S = max(|W|)/127. Quantize each weight as q = clamp(round(w/S), −127, 127), then reconstruct it as ŵ = qS. If the tensor is all zeros, use a nonzero fallback scale such as 1.0. section 2.2 section 3.2

Listing 3.9 Per-tensor symmetric quantizer
class PerTensorQuantizer:
    def __init__(self, bits=8):
        self.q_max = (1 << (bits - 1)) - 1                       #A
        self.q_min = -self.q_max
        self.scale = None
    
    def calibrate(self, tensor):
        abs_max = tensor.abs().max()
        self.scale = abs_max.item() / self.q_max if abs_max > 0 else 1.0
 
    def quantize(self, tensor):
        scaled = tensor / self.scale
        rounded = torch.round(scaled)
        return torch.clamp(rounded, self.q_min, self.q_max).to(torch.int8)
    
    def dequantize(self, q_tensor):
        return q_tensor.float() * self.scale
 
    def quantize_dequantize(self, tensor):
        self.calibrate(tensor)
        return self.dequantize(self.quantize(tensor))            #B
 

The PyTorch quantizer calibrates one symmetric scale, clamps INT8 values, and reconstructs the tensor. section 3.2

Outliers and channel utilization

In ResNet-18’s first convolution, a single INT8 scale gives a reconstruction MSE of 0.00000466, but mean channel utilization is only 39.89% of the integer range. Per-channel scaling reduces MSE to 0.00000129, or 3.61 times lower. Aggregate error alone therefore hides how much precision smaller channels lose. section 3.2

The same problem affects activations: one high-magnitude token can determine the scale for an entire batch. Per-token scaling limits that interaction, while an outlier-aware mixed-precision path can isolate extreme feature dimensions. FP8 also needs scaling; block scaling confines an outlier’s effect to its block, at the cost of additional scale storage. section 7.2 section 8.2.2

Four-bit quality versus storage

For OPT-6.7B linear weights quantized to FP4 and dequantized to BF16 for matmul, per-tensor scaling yields WikiText-2 perplexity 26.09 at an estimated 3.11 GB, versus 16.26 for the 12.40 GB BF16 baseline. FP4 block-16 scaling with FP32 scales recovers perplexity to 16.49 at 4.66 GB. These results measure quality and estimated storage, not native FP4 throughput. section 8.3.4

Use per-tensor scaling when its measured error and model quality are acceptable for the deployment. Compare against per-channel or block scaling when magnitudes vary sharply, especially at low bit widths. Check runtime and hardware support before choosing finer granularity: extra scales consume storage and may require different kernel support. section 3.0 section 2.2 section 8.3.2

How Quantization and Fast Inference covers it

How deep it goes
Derives symmetric INT8 calibration and reconstruction, implements a quantizer, and compares error and storage against finer-grained scaling.
Examples
Measures ResNet-18 convolution weight reconstruction and OPT-6.7B FP4 perplexity under different scaling granularities.
What you can do after
Build it from scratch by calibrating a shared scale, quantizing and reconstructing values, and measuring reconstruction error.
How it connects
Connects affine quantization and outlier sensitivity to per-channel, per-token, blockwise, and mixed-precision quantization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.