liveBook · Manning

GPTQ

GPTQ is a post-training weight-quantization method that uses calibration data and approximate second-order information to reduce neural-network precision while limiting quantization error.

How 2 Manning books explain it

Quantization and Fast Inference

GPTQ is a post-training quantization algorithm for compressing transformer model weights with controlled accuracy loss.

How it is used
Compresses transformer weights after training while controlling layer-output reconstruction error.
How deep it goes
Derives the Hessian-based objective and details calibration, blockwise error compensation, group-size tradeoffs, and deployment configuration.
Examples
Measures OPT-6.7B layer reconstruction error, group-size tradeoffs, WikiText-2 perplexity, and model memory against FP16 and round-to-nearest.
What you can do after
Apply it via GPTQModel to calibrate, quantize, and save a 4-bit model with a specified group size and damping.

Read how Quantization and Fast Inference explains GPTQ →

Domain-Specific Small Language Models

A one-shot post-training weight quantization method that uses approximate second-order information to compress generative pretrained transformers with minimal accuracy loss.

How it is used
Compresses pretrained GPT weights for lower-memory inference while aiming to preserve accuracy.
How deep it goes
Explains the one-shot method and demonstrates quantization, while deferring mathematical and low-level implementation details.
Examples
Uses GPT-2 XL, WikiText samples, a 4-bit configuration, and Tesla T4 inference.
What you can do after
Apply it via AutoGPTQ to quantize, save, reload, and generate with a 4-bit GPT-2 XL model.

Read how Domain-Specific Small Language Models explains GPTQ →

Also covered in

What to know first

Related concepts

Questions about GPTQ

What is GPTQ?

GPTQ is a post-training weight-quantization method that uses calibration data and approximate second-order information to reduce neural-network precision while limiting quantization error.

Which Manning books explain GPTQ?

Quantization and Fast Inference and Domain-Specific Small Language Models.

What should I know before learning GPTQ?

Manning books that teach GPTQ usually introduce float16, key-value cache, quantization and perplexity first.

What is the difference between GPTQ and AWQ?

GPTQ is a post-training weight-quantization method that uses calibration data and approximate second-order information to reduce neural-network precision while limiting quantization error. AWQ is a post-training weight-quantization method that uses activation statistics to identify and protect salient weights, reducing model-size and precision while limiting quality loss.

What is GPTQ a kind of?

GPTQ is a kind of post-training quantization. Post-training quantization converts a trained model’s parameters or computations to lower-precision representations without further model training.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.