GPTQ is a post-training method for compressing transformer weights, commonly to 4 bits. It uses calibration activations to compensate for rounding error, aiming to preserve layer outputs rather than merely minimize differences between original and quantized weights. section 7.3 section 7.7
For a linear layer Y = WX, GPTQ targets the squared output error ||WX − QX||², where Q is the quantized weight matrix and X contains calibration activations. Its Hessian, H = XXᵀ, measures sensitivity to weight perturbations: large diagonal entries indicate input dimensions whose rounding errors can strongly affect the output. section 7.3
GPTQ processes weight columns left to right. After quantizing a column, it uses inverse-Hessian information to distribute the rounding error across columns not yet quantized. Processing columns in blocks and using a Cholesky factorization makes the procedure practical; damping the Hessian supports numerical stability. section 7.3 section 7.7
Figure 7.7 GPTQ algorithm flow. Calibration activations X produce the Hessian H = XXT, which is Cholesky-factored for numerical stability. The weight matrix W is processed in blocks of B=128 columns. Within each block, each column is quantized to INT4, and the rounding error is distributed across remaining columns using the inverse Hessian. The output is the quantized weight matrix Q plus per-group scales.

Follow the path from calibration activations to blockwise INT4 weights and per-group scales. section 7.3
OPT-6.7B reconstruction and perplexity
For layer 0 fc1, 4-bit GPTQ reduced reconstruction MSE from 0.0224 with round-to-nearest to 0.0002, a 112× improvement. In a separate WikiText-2 evaluation, 4-bit GPTQ with group size 128 reached perplexity 16.15 versus 16.31 for FP16 and loaded in 3,626 MB versus 12,700 MB. The 0.16-point perplexity difference was within measurement noise at that evaluation length. section 7.3
Group size trades reconstruction error against scale and zero-point storage. In the measured example, group size 128 sits at the knee of the error-versus-storage curve; groups of 64 or 32 give diminishing error reductions at higher storage cost. Validate quality on the intended distribution: the reported no-measurable-loss result assumes a matched calibration distribution. section 7.3 section 7.7
Listing 7.8 GPTQ production deployment with gptqmodel
from gptqmodel import GPTQModel, QuantizeConfig
quantize_config = QuantizeConfig(
bits=4, #A
group_size=128, #B
damp_percent=0.01, #C
desc_act=False, #D
)
model = GPTQModel.load("facebook/opt-6.7b", quantize_config)
model.quantize(calib_data) #E
model.save_quantized("./opt-6.7b-gptq-4bit") #F
Use GPTQModel to quantize and save a model with 4-bit weights, group size 128, and 0.01 damping. section 7.3

