liveBook · Manning

Activation magnitude

As explained in Quantization and Fast Inference

Activation magnitude is the absolute size of a neural-network activation, considered per feature or channel.

Activation magnitude is the absolute size of an activation, considered per feature or channel. For weight quantization, it helps identify channels where rounding errors can have a large effect on a layer’s output. section 7.4

A small weight can carry a large signal

In y = Wx, a weight of 0.001 multiplied by an activation of 1,000 contributes 1.0. Rounding that weight to zero removes the contribution, even though the weight itself is small. section 7.4

A practical importance score is the mean absolute activation for each channel across calibration samples. Activation-aware weight quantization (AWQ) uses these scores to search for per-channel scales that reduce reconstruction error after quantization. Scaling weights and inversely scaling activations preserve the unquantized computation; the scales change where quantization error falls. section 7.4

Listing 7.9 Computing per-channel activation importance
def compute_activation_scales(X):
    """Per-channel importance = mean(|x|) across calibration samples."""
    act_scales = X.abs().mean(dim=0)              #A
    return act_scales                             #B
 

Compute channel scores by taking absolute values and averaging across calibration samples. section 7.4

For OPT-6.7B’s first fc1 layer, activation magnitude correlated with per-channel quantization error at r = 0.949, versus −0.115 for weight magnitude. Activation ranking identified 38 of the 50 most damaging channels; weight ranking identified none. These results are specific to that measurement. section 7.4

How Quantization and Fast Inference covers it

How deep it goes
Explains mean-absolute channel scoring, scaling, reconstruction-error search, and empirical comparisons with weight magnitude.
Examples
Compares channel rankings on OPT-6.7B layer 0 fc1 using 128 WikiText-2 calibration sequences.
What you can do after
Build per-channel activation importance scores from scratch using calibration samples.
How it connects
Connects activation magnitude to weight quantization, calibration, per-channel scaling, and reconstruction error.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.