liveBook · Manning

LLM.int8()

As explained in Quantization and Fast Inference

LLM.int8() is a quantization method for large language models that uses 8-bit computation while handling activation outliers with higher precision.

LLM.int8() reduces large-model memory by routing rare activation outliers through FP16 while computing the remaining dimensions in INT8. The split is dynamic on each forward pass and needs no calibration data, retraining, or model modification. section 7.2 section 7.7

The matrix multiplication decomposition proceeds as follows:

  • Detect outlier feature dimensions from activation magnitudes. section 7.2
  • Split activations and weights by feature dimension. section 7.2
  • Compute outlier columns in full precision and normal columns in INT8, then sum both outputs. section 7.2

OPT-6.7B memory and perplexity

On WikiText-2, LLM.int8() matches the FP16 perplexity of 16.31 while reducing GPU memory from 12.4 GB to 6.4 GB; naive INT8 reaches 48.35 perplexity. section 7.2

Listing 7.4 Mixed-precision decomposition (the LLM.int8() algorithm)
def mixed_precision_matmul(X, W, threshold=6.0):
    # Step 1: Identify outlier feature dimensions
    outlier_mask = X.abs().max(dim=0).values > threshold          #A
    outlier_dims = outlier_mask.nonzero(as_tuple=True)[0]
    normal_dims = (~outlier_mask).nonzero(as_tuple=True)[0]
 
    # Step 2: Split into two paths
    X_outlier = X[:, outlier_dims].float()                        #B
    W_outlier = W[:, outlier_dims].float()
    X_normal = X[:, normal_dims].float()
    W_normal = W[:, normal_dims]
 
    # Step 3a: Full-precision matmul for outlier dimensions
    out_outlier = X_outlier @ W_outlier.T                         #C
 
    # Step 3b: INT8 matmul for normal dimensions
    W_q, W_s = quantize_absmax_int8(W_normal)                     #D
    X_scales = X_normal.abs().amax(dim=1, keepdim=True) / 127.0
    X_q = torch.round(X_normal / X_scales).clamp(-128, 127)
    out_normal = (X_q.float() @ W_q.float().T) * (X_scales * W_s.T)
 
    # Step 4: Sum the two paths
    return out_outlier + out_normal                                #E
 

Inspect the mixed-precision matmul implementation, including threshold-based outlier detection and separate precision paths. section 7.2

How Quantization and Fast Inference covers it

How deep it goes
Derives the four-step decomposition, per-channel weight and per-token activation scaling, and threshold tradeoffs.
Examples
Uses OPT-6.7B measurements, a threshold sweep, and bitsandbytes deployment configuration.
What you can do after
Apply LLM.int8() through Transformers and bitsandbytes using an 8-bit loading option and configurable outlier threshold.
How it connects
Connects activation outliers and INT8 quantization to FP16 computation, and contrasts the method with GPTQ and AWQ.

Also explained in

Compare how 2 books explain LLM.int8() →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.