Mixed precision reduces LLM weight-and-activation quantization error by sending outlier feature dimensions through a higher-precision path and the remaining dimensions through INT8. section 7.2
The decomposition works as follows:
- Flag activation dimensions whose maximum absolute value exceeds a threshold, such as 6.0. section 7.2
- Split activations and weights along those dimensions; compute the outlier slice at higher precision and the remaining slice with INT8, then sum the outputs. section 7.2
Listing 7.4 Mixed-precision decomposition (the LLM.int8() algorithm)
def mixed_precision_matmul(X, W, threshold=6.0):
# Step 1: Identify outlier feature dimensions
outlier_mask = X.abs().max(dim=0).values > threshold #A
outlier_dims = outlier_mask.nonzero(as_tuple=True)[0]
normal_dims = (~outlier_mask).nonzero(as_tuple=True)[0]
# Step 2: Split into two paths
X_outlier = X[:, outlier_dims].float() #B
W_outlier = W[:, outlier_dims].float()
X_normal = X[:, normal_dims].float()
W_normal = W[:, normal_dims]
# Step 3a: Full-precision matmul for outlier dimensions
out_outlier = X_outlier @ W_outlier.T #C
# Step 3b: INT8 matmul for normal dimensions
W_q, W_s = quantize_absmax_int8(W_normal) #D
X_scales = X_normal.abs().amax(dim=1, keepdim=True) / 127.0
X_q = torch.round(X_normal / X_scales).clamp(-128, 127)
out_normal = (X_q.float() @ W_q.float().T) * (X_scales * W_s.T)
# Step 4: Sum the two paths
return out_outlier + out_normal #E
Inspect the PyTorch implementation of outlier detection and the two matmul paths. section 7.2
OPT-6.7B tradeoff
On WikiText-2, LLM.int8() matches the FP16 perplexity of 16.31 while using 6.4 GB of GPU memory, versus 12.4 GB for FP16. Naive INT8 reaches perplexity 48.35. section 7.2





