liveBook · Manning

GPTQ

As explained in Quantization and Fast Inference

GPTQ is a post-training quantization algorithm for compressing transformer model weights with controlled accuracy loss.

GPTQ is a post-training method for compressing transformer weights, commonly to 4 bits. It uses calibration activations to compensate for rounding error, aiming to preserve layer outputs rather than merely minimize differences between original and quantized weights. section 7.3 section 7.7

For a linear layer Y = WX, GPTQ targets the squared output error ||WX − QX||², where Q is the quantized weight matrix and X contains calibration activations. Its Hessian, H = XXᵀ, measures sensitivity to weight perturbations: large diagonal entries indicate input dimensions whose rounding errors can strongly affect the output. section 7.3

GPTQ processes weight columns left to right. After quantizing a column, it uses inverse-Hessian information to distribute the rounding error across columns not yet quantized. Processing columns in blocks and using a Cholesky factorization makes the procedure practical; damping the Hessian supports numerical stability. section 7.3 section 7.7

Figure 7.7 GPTQ algorithm flow. Calibration activations X produce the Hessian H = XXT, which is Cholesky-factored for numerical stability. The weight matrix W is processed in blocks of B=128 columns. Within each block, each column is quantized to INT4, and the rounding error is distributed across remaining columns using the inverse Hessian. The output is the quantized weight matrix Q plus per-group scales.

Follow the path from calibration activations to blockwise INT4 weights and per-group scales. section 7.3

OPT-6.7B reconstruction and perplexity

For layer 0 fc1, 4-bit GPTQ reduced reconstruction MSE from 0.0224 with round-to-nearest to 0.0002, a 112× improvement. In a separate WikiText-2 evaluation, 4-bit GPTQ with group size 128 reached perplexity 16.15 versus 16.31 for FP16 and loaded in 3,626 MB versus 12,700 MB. The 0.16-point perplexity difference was within measurement noise at that evaluation length. section 7.3

Group size trades reconstruction error against scale and zero-point storage. In the measured example, group size 128 sits at the knee of the error-versus-storage curve; groups of 64 or 32 give diminishing error reductions at higher storage cost. Validate quality on the intended distribution: the reported no-measurable-loss result assumes a matched calibration distribution. section 7.3 section 7.7

Listing 7.8 GPTQ production deployment with gptqmodel
from gptqmodel import GPTQModel, QuantizeConfig
 
quantize_config = QuantizeConfig(
    bits=4,                                                 #A
    group_size=128,                                         #B
    damp_percent=0.01,                                      #C
    desc_act=False,                                         #D
)
 
model = GPTQModel.load("facebook/opt-6.7b", quantize_config)
model.quantize(calib_data)                                  #E
model.save_quantized("./opt-6.7b-gptq-4bit")                #F
 

Use GPTQModel to quantize and save a model with 4-bit weights, group size 128, and 0.01 damping. section 7.3

How Quantization and Fast Inference covers it

How deep it goes
Derives the Hessian-based objective and details calibration, blockwise error compensation, group-size tradeoffs, and deployment configuration.
Examples
Measures OPT-6.7B layer reconstruction error, group-size tradeoffs, WikiText-2 perplexity, and model memory against FP16 and round-to-nearest.
What you can do after
Apply it via GPTQModel to calibrate, quantize, and save a 4-bit model with a specified group size and damping.
How it connects
Connects post-training quantization to Hessian sensitivity, reconstruction error, calibration, and round-to-nearest quantization.

Also explained in

Compare how 2 books explain GPTQ →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.