GPTQ
GPTQ is a post-training weight-quantization method that uses calibration data and approximate second-order information to reduce neural-network precision while limiting quantization error.
Quantization and Fast Inference
GPTQ is a post-training quantization algorithm for compressing transformer model weights with controlled accuracy loss.
- How it is used
- Compresses transformer weights after training while controlling layer-output reconstruction error.
- How deep it goes
- Derives the Hessian-based objective and details calibration, blockwise error compensation, group-size tradeoffs, and deployment configuration.
- Examples
- Measures OPT-6.7B layer reconstruction error, group-size tradeoffs, WikiText-2 perplexity, and model memory against FP16 and round-to-nearest.
- What you can do after
- Apply it via GPTQModel to calibrate, quantize, and save a 4-bit model with a specified group size and damping.
Domain-Specific Small Language Models
A one-shot post-training weight quantization method that uses approximate second-order information to compress generative pretrained transformers with minimal accuracy loss.
- How it is used
- Compresses pretrained GPT weights for lower-memory inference while aiming to preserve accuracy.
- How deep it goes
- Explains the one-shot method and demonstrates quantization, while deferring mathematical and low-level implementation details.
- Examples
- Uses GPT-2 XL, WikiText samples, a 4-bit configuration, and Tesla T4 inference.
- What you can do after
- Apply it via AutoGPTQ to quantize, save, reload, and generate with a 4-bit GPT-2 XL model.
Read how Domain-Specific Small Language Models explains GPTQ →
Also covered in
- Rearchitecting LLMsIn early access (MEAP)
- CUDA for LLMs
What to know first
- float16float16 is a 16-bit floating-point format, commonly IEEE 754 binary16, with less precision and range than 32-bit floating-point formats.
- key-value cacheA key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.
- quantizationModel quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
- perplexityPerplexity is a language-model metric, typically the exponential of average cross-entropy, that measures how well a model predicts a sequence, with lower values indicating better predictive fit.
Related concepts
- error compensationuses · Error compensation redistributes or offsets quantization errors across values to reduce overall reconstruction error.
- int4 quantizationuses · INT4 quantization represents numerical values using 4-bit integers, reducing storage requirements while limiting numerical precision.
- AWQcontrasts with · AWQ is a post-training weight-quantization method that uses activation statistics to identify and protect salient weights, reducing model-size and precision while limiting quality loss.
- Hessian matrixuses · A Hessian matrix is the square matrix of second-order partial derivatives of a scalar-valued function of multiple variables.
- int4uses · INT4 is a four-bit integer numerical format used for low-precision representation and computation.
- post-training quantizationkind of · Post-training quantization converts a trained model’s parameters or computations to lower-precision representations without further model training.
Questions about GPTQ
What is GPTQ?
GPTQ is a post-training weight-quantization method that uses calibration data and approximate second-order information to reduce neural-network precision while limiting quantization error.
Which Manning books explain GPTQ?
Quantization and Fast Inference and Domain-Specific Small Language Models.
What should I know before learning GPTQ?
Manning books that teach GPTQ usually introduce float16, key-value cache, quantization and perplexity first.
What is the difference between GPTQ and AWQ?
GPTQ is a post-training weight-quantization method that uses calibration data and approximate second-order information to reduce neural-network precision while limiting quantization error. AWQ is a post-training weight-quantization method that uses activation statistics to identify and protect salient weights, reducing model-size and precision while limiting quality loss.
What is GPTQ a kind of?
GPTQ is a kind of post-training quantization. Post-training quantization converts a trained model’s parameters or computations to lower-precision representations without further model training.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

