Per-tensor quantization uses one set of quantization parameters for every value in a tensor. It minimizes scale metadata and provides a simple baseline, but an outlier can set a coarse resolution for all other values. The tradeoff depends on the tensor’s distribution, precision, and target hardware. section 3.0 section 2.2
In affine quantization, the entire tensor shares a scale S and zero point Z. For symmetric INT8 weight quantization, Z is zero and S = max(|W|)/127. Quantize each weight as q = clamp(round(w/S), −127, 127), then reconstruct it as ŵ = qS. If the tensor is all zeros, use a nonzero fallback scale such as 1.0. section 2.2 section 3.2
Listing 3.9 Per-tensor symmetric quantizer
class PerTensorQuantizer:
def __init__(self, bits=8):
self.q_max = (1 << (bits - 1)) - 1 #A
self.q_min = -self.q_max
self.scale = None
def calibrate(self, tensor):
abs_max = tensor.abs().max()
self.scale = abs_max.item() / self.q_max if abs_max > 0 else 1.0
def quantize(self, tensor):
scaled = tensor / self.scale
rounded = torch.round(scaled)
return torch.clamp(rounded, self.q_min, self.q_max).to(torch.int8)
def dequantize(self, q_tensor):
return q_tensor.float() * self.scale
def quantize_dequantize(self, tensor):
self.calibrate(tensor)
return self.dequantize(self.quantize(tensor)) #B
The PyTorch quantizer calibrates one symmetric scale, clamps INT8 values, and reconstructs the tensor. section 3.2
Outliers and channel utilization
In ResNet-18’s first convolution, a single INT8 scale gives a reconstruction MSE of 0.00000466, but mean channel utilization is only 39.89% of the integer range. Per-channel scaling reduces MSE to 0.00000129, or 3.61 times lower. Aggregate error alone therefore hides how much precision smaller channels lose. section 3.2
The same problem affects activations: one high-magnitude token can determine the scale for an entire batch. Per-token scaling limits that interaction, while an outlier-aware mixed-precision path can isolate extreme feature dimensions. FP8 also needs scaling; block scaling confines an outlier’s effect to its block, at the cost of additional scale storage. section 7.2 section 8.2.2
Four-bit quality versus storage
For OPT-6.7B linear weights quantized to FP4 and dequantized to BF16 for matmul, per-tensor scaling yields WikiText-2 perplexity 26.09 at an estimated 3.11 GB, versus 16.26 for the 12.40 GB BF16 baseline. FP4 block-16 scaling with FP32 scales recovers perplexity to 16.49 at 4.66 GB. These results measure quality and estimated storage, not native FP4 throughput. section 8.3.4
Use per-tensor scaling when its measured error and model quality are acceptable for the deployment. Compare against per-channel or block scaling when magnitudes vary sharply, especially at low bit widths. Check runtime and hardware support before choosing finer granularity: extra scales consume storage and may require different kernel support. section 3.0 section 2.2 section 8.3.2
