A gradient carries the learning signal used to update model weights. Quantization disrupts that signal: rounding has derivative zero almost everywhere, so true quantized values can leave weights with no gradient and no update. section 5.2
Quantization-aware training preserves gradients while simulating quantized values:
- Fake quantization rounds and rescales values in the forward pass while retaining full-precision weights for gradient computation. section 5.2
- The straight-through estimator (STE) approximates the rounding derivative as 1, passing gradients backward; a clipped STE instead zeros gradients for values outside the quantizable range. section 5.2
- Per-channel scales can reduce clipping compared with one scale per tensor, helping more gradients survive. section 5.4.1
The basic STE treats the rounding operation's derivative as approximately 1, allowing gradients to pass through. section 5.2
INT4 gradient survival in ResNet-18 conv1
With per-tensor quantization, an average of 67.4% of gradients are clipped per channel, and five channels lose all gradients. Per-channel quantization reduces the average to 31.5%, a 2.10× improvement in gradient survival. section 5.4.1


























