LLM.int8() reduces large-model memory by routing rare activation outliers through FP16 while computing the remaining dimensions in INT8. The split is dynamic on each forward pass and needs no calibration data, retraining, or model modification. section 7.2 section 7.7
The matrix multiplication decomposition proceeds as follows:
- Detect outlier feature dimensions from activation magnitudes. section 7.2
- Split activations and weights by feature dimension. section 7.2
- Compute outlier columns in full precision and normal columns in INT8, then sum both outputs. section 7.2
OPT-6.7B memory and perplexity
On WikiText-2, LLM.int8() matches the FP16 perplexity of 16.31 while reducing GPU memory from 12.4 GB to 6.4 GB; naive INT8 reaches 48.35 perplexity. section 7.2
Listing 7.4 Mixed-precision decomposition (the LLM.int8() algorithm)
def mixed_precision_matmul(X, W, threshold=6.0):
# Step 1: Identify outlier feature dimensions
outlier_mask = X.abs().max(dim=0).values > threshold #A
outlier_dims = outlier_mask.nonzero(as_tuple=True)[0]
normal_dims = (~outlier_mask).nonzero(as_tuple=True)[0]
# Step 2: Split into two paths
X_outlier = X[:, outlier_dims].float() #B
W_outlier = W[:, outlier_dims].float()
X_normal = X[:, normal_dims].float()
W_normal = W[:, normal_dims]
# Step 3a: Full-precision matmul for outlier dimensions
out_outlier = X_outlier @ W_outlier.T #C
# Step 3b: INT8 matmul for normal dimensions
W_q, W_s = quantize_absmax_int8(W_normal) #D
X_scales = X_normal.abs().amax(dim=1, keepdim=True) / 127.0
X_q = torch.round(X_normal / X_scales).clamp(-128, 127)
out_normal = (X_q.float() @ W_q.float().T) * (X_scales * W_s.T)
# Step 4: Sum the two paths
return out_outlier + out_normal #E
Inspect the mixed-precision matmul implementation, including threshold-based outlier detection and separate precision paths. section 7.2

