How deeply does Domain-Specific Small Language Models cover GPTQ?

Defines the method, gives its rationale, and walks through AutoGPTQ usage, but defers low-level math and implementation details.

GPTQ

As explained in Domain-Specific Small Language Models

A one-shot post-training weight quantization method that uses approximate second-order information to compress generative pretrained transformers with minimal accuracy loss.

GPTQ is a one-shot post-training weight quantization method for generative pretrained transformers. It uses approximate second-order information to compress models to 3 or 4 bits per weight with negligible accuracy loss. section 6.4.1 section 6.5

It is used to shrink large models enough for practical inference on limited hardware, while keeping performance close to the uncompressed baseline. section 6.4.1 section 6.5

Typical treatment centers on these implementation steps:

Configure 4-bit quantization parameters such as bit width and group size. section 6.4.1
Load the pretrained model and a calibration dataset. section 6.4.1
Run quantization, save the quantized model, and reload it for inference. section 6.4.1

AutoGPTQ workflow

The example quantizes GPT-2 XL with AutoGPTQ, uses Wikitext-2 for calibration, saves a 4-bit model, reloads it on CUDA, and generates text from the prompt “Auto-GPTQ is”. section 6.4.1

How Domain-Specific Small Language Models covers it

Cover of Domain-Specific Small Language Models

liveBook manning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

post-training quantizationkind of · Post-training quantization is the conversion of a trained model’s parameters or computations to lower-precision representations without additional training.
int4 quantizationuses · INT4 quantization is a technique that represents model weights, activations, or computations using four-bit integer values to reduce storage and computational requirements.
Hessian matrixuses · A Hessian matrix is the square matrix of second-order partial derivatives of a scalar-valued multivariable function, describing its local curvature.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.