liveBook · Manning

Error compensation

As explained in Quantization and Fast Inference

Error compensation redistributes quantization error from already processed values into remaining values to reduce reconstruction error.

In GPTQ, error compensation reduces reconstruction error during column-wise weight quantization. After rounding a column, it redistributes the rounding error across columns still to be processed, weighted by their inverse-Hessian relationships. section 7.3

This preserves information that independent rounding discards: strongly related dimensions receive larger corrections, while weakly related dimensions receive smaller ones. Calibration activations provide the Hessian used to guide the corrections. section 7.3

Budget-rounding analogy

If rounding one department’s budget down creates a shortfall, add a weighted correction to later budgets before rounding them. The analogy illustrates how GPTQ spreads error instead of accepting each rounding loss independently. section 7.3

Figure 7.7 GPTQ algorithm flow. Calibration activations X produce the Hessian H = XXT, which is Cholesky-factored for numerical stability. The weight matrix W is processed in blocks of B=128 columns. Within each block, each column is quantized to INT4, and the rounding error is distributed across remaining columns using the inverse Hessian. The output is the quantized weight matrix Q plus per-group scales.

Shows the GPTQ workflow from calibration activations and Hessian construction through blocked INT4 quantization and inverse-Hessian error redistribution. section 7.3

How Quantization and Fast Inference covers it

How deep it goes
Explains inverse-Hessian-weighted error redistribution across columns and describes blocked processing.
Examples
Illustrates weighted corrections with a budget-rounding analogy and reports GPTQ reconstruction-error comparisons.
What it assumes
Assumes weight quantization and calibration activations, from which it computes the Hessian.
What you can do after
Apply it through GPTQ to distribute column-rounding errors using calibration-derived Hessian information.
How it connects
Connects GPTQ error compensation to Hessians, calibration activations, and weight quantization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.