liveBook · Manning

Equivalent transformation

As explained in Quantization and Fast Inference

An equivalent transformation changes internal model representations while preserving the model's output.

Equivalent transformation lets AWQ protect activation-salient weight channels before quantization without changing the layer’s output beforehand. Scale each weight column by s and apply the inverse scale to the matching activation channel; then quantize the scaled weights. At inference, fold the inverse scaling into the preceding operator, typically LayerNorm. section 7.4

Equation 7.3
Y=W.X=(W.diag(s)).(diag(s)-1.X)=W'X'

The identity shows how weight and activation scaling cancel in the matrix product. section 7.4

Quantization error tradeoff

For OPT-6.7B layer 0 fc1 at 4-bit group size 128, AWQ reduces reconstruction error 17.1× versus round-to-nearest quantization. The inverse scale is folded into LayerNorm, avoiding runtime overhead. section 7.4

How Quantization and Fast Inference covers it

How deep it goes
Derives the scaling identity, explains quantization effects, and describes folding inverse scales into LayerNorm.
Examples
Compares 4-bit OPT-6.7B reconstruction error, reporting a 17.1× reduction with group size 128.
What you can do after
Apply AWQ scaling and quantization with AutoAWQ using calibration data and configurable bit width and group size.
How it connects
Links activation-aware weight quantization with calibration data, group-wise quantization, and LayerNorm.

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.