Activation magnitude is the absolute size of an activation, considered per feature or channel. For weight quantization, it helps identify channels where rounding errors can have a large effect on a layer’s output. section 7.4
A small weight can carry a large signal
In y = Wx, a weight of 0.001 multiplied by an activation of 1,000 contributes 1.0. Rounding that weight to zero removes the contribution, even though the weight itself is small. section 7.4
A practical importance score is the mean absolute activation for each channel across calibration samples. Activation-aware weight quantization (AWQ) uses these scores to search for per-channel scales that reduce reconstruction error after quantization. Scaling weights and inversely scaling activations preserve the unquantized computation; the scales change where quantization error falls. section 7.4
Listing 7.9 Computing per-channel activation importance
def compute_activation_scales(X):
"""Per-channel importance = mean(|x|) across calibration samples."""
act_scales = X.abs().mean(dim=0) #A
return act_scales #B
Compute channel scores by taking absolute values and averaging across calibration samples. section 7.4
For OPT-6.7B’s first fc1 layer, activation magnitude correlated with per-channel quantization error at r = 0.949, versus −0.115 for weight magnitude. Activation ranking identified 38 of the 50 most damaging channels; weight ranking identified none. These results are specific to that measurement. section 7.4
