Per-channel quantization assigns separate quantization parameters to each tensor channel, preventing a large value in one channel from setting a coarse quantization grid for all others. For weights, channels usually correspond to outputs; affine quantization can use a separate scale and zero point for each channel. section 3.2 section 2.2
For symmetric INT8 weights W shaped [out_features, in_features], calculate S_c = max(|W_c|)/127 for each output channel c. Quantize with q_c = clamp(round(W_c/S_c), -127, 127) and reconstruct with Ŵ_c = q_c × S_c. Broadcast the scale vector across the remaining dimensions; use a nonzero fallback scale for an all-zero channel. section 3.2 section 5.4.1
Resolution and reconstruction error
For ResNet-18’s first convolution, a single tensor-wide scale leaves channels using 39.89% of the integer range on average. Per-channel INT8 reduces weight reconstruction MSE from 0.00000466 to 0.00000129, a 3.61× improvement. This is a weight-error comparison, not a guarantee of the same improvement in model accuracy. section 3.2
The extra scales cost memory and must be supported efficiently by the target runtime. With FP32 scales and INT8 weights, scale storage is 2.72% of weight storage for ResNet-18 conv1 but 0.10% for a Llama-7B query projection. Check kernel support and measure latency rather than assuming improved accuracy implies faster inference. section 3.2 section 2.2
In quantization-aware training, fake quantization applies each channel’s grid in the forward pass and uses a straight-through estimator for gradients. Recompute scales as weights evolve, but detach scale computation from gradients. In one INT4 ResNet-18 conv1 experiment, the estimator masked 67.4% of gradients per channel on average with a tensor-wide scale, versus 31.5% with per-channel scales; the mask also excludes quantized boundary values. section 5.4.1
Listing 5.11 Per-channel fake quantization for QAT
class FakeQuantizePerChannel(torch.autograd.Function):
"""Per-channel fake quantization with STE."""
@staticmethod
def forward(ctx, weight, scales, q_min, q_max, axis):
shape = [1] * weight.dim()
shape[axis] = -1 #A
scales_bc = scales.view(shape)
w_q = torch.clamp(torch.round(weight / scales_bc),
q_min, q_max)
w_dq = w_q * scales_bc
mask = (w_q > q_min) & (w_q < q_max) #B
ctx.save_for_backward(mask)
return w_dq
@staticmethod
def backward(ctx, grad_output):
mask, = ctx.saved_tensors
return grad_output * mask.float(), None, None, None, None #CInspect scale broadcasting and the element-wise straight-through gradient mask in a per-channel fake quantizer. section 5.4.1
Choose granularity against the deployment target:
- Use output-channel axis 0 for convolution and linear weights; test accuracy and runtime compatibility on the intended model and hardware. section 5.4.2 section 3.7
- Consider group-wise quantization for INT4 weights when per-channel error remains too high; smaller groups add scale metadata and impose layout and kernel constraints. section 3.7 section 2.2
- Treat activation and KV-cache granularity separately: activation statistics vary with input, while cache keys and values can favor different granularities. section 3.0 section 3.7

