liveBook · Manning

Mixed precision

As explained in Quantization and Fast Inference

Mixed precision uses different numerical precisions for different tensors or operations within a computation.

Mixed precision reduces LLM weight-and-activation quantization error by sending outlier feature dimensions through a higher-precision path and the remaining dimensions through INT8. section 7.2

The decomposition works as follows:

  • Flag activation dimensions whose maximum absolute value exceeds a threshold, such as 6.0. section 7.2
  • Split activations and weights along those dimensions; compute the outlier slice at higher precision and the remaining slice with INT8, then sum the outputs. section 7.2
Listing 7.4 Mixed-precision decomposition (the LLM.int8() algorithm)
def mixed_precision_matmul(X, W, threshold=6.0):
    # Step 1: Identify outlier feature dimensions
    outlier_mask = X.abs().max(dim=0).values > threshold          #A
    outlier_dims = outlier_mask.nonzero(as_tuple=True)[0]
    normal_dims = (~outlier_mask).nonzero(as_tuple=True)[0]
 
    # Step 2: Split into two paths
    X_outlier = X[:, outlier_dims].float()                        #B
    W_outlier = W[:, outlier_dims].float()
    X_normal = X[:, normal_dims].float()
    W_normal = W[:, normal_dims]
 
    # Step 3a: Full-precision matmul for outlier dimensions
    out_outlier = X_outlier @ W_outlier.T                         #C
 
    # Step 3b: INT8 matmul for normal dimensions
    W_q, W_s = quantize_absmax_int8(W_normal)                     #D
    X_scales = X_normal.abs().amax(dim=1, keepdim=True) / 127.0
    X_q = torch.round(X_normal / X_scales).clamp(-128, 127)
    out_normal = (X_q.float() @ W_q.float().T) * (X_scales * W_s.T)
 
    # Step 4: Sum the two paths
    return out_outlier + out_normal                                #E
 

Inspect the PyTorch implementation of outlier detection and the two matmul paths. section 7.2

OPT-6.7B tradeoff

On WikiText-2, LLM.int8() matches the FP16 perplexity of 16.31 while using 6.4 GB of GPU memory, versus 12.4 GB for FP16. Naive INT8 reaches perplexity 48.35. section 7.2

How Quantization and Fast Inference covers it

How deep it goes
Explains threshold-based outlier routing, split matmuls, reconstruction error, and threshold sweeps.
Examples
Compares OPT-6.7B reconstruction error and WikiText-2 perplexity, memory, and threshold settings.
What you can do after
Apply LLM.int8() with bitsandbytes by configuring 8-bit loading and an outlier threshold.
How it connects
Connects INT8 quantization with activation outliers, per-channel quantization, and LLM.int8().

Also explained in

Compare how 6 books explain mixed precision →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.