Fake quantization simulates low-precision deployment during training: the forward pass quantizes and dequantizes values, while full-precision weights remain available for gradient updates. This lets a model adapt to quantization noise instead of merely tolerating it. section 5.0 section 5.2.1
The scale maps values to a quantization grid and back; rounding creates the gradient problem addressed by the straight-through estimator. section 5.2
In a QAT forward pass:
- Round and clip values to the configured integer range, then dequantize them for computation. section 5.2
- Use a straight-through estimator (STE) to pass gradients through rounding; the clipped variant zeros gradients for saturated values. section 5.2
- Per-channel scales can preserve gradient flow across channels; per-tensor scales may leave small-magnitude channels clipped. section 5.4
Quantizing with a scale of 0.5
The values [0.3, 0.7, 1.2, 1.8] become [0.5, 0.5, 1.0, 2.0]. Ordinary rounding produces zero gradients; an STE supplies a learning signal during backpropagation. section 5.2
For linear and convolution layers, fake-quantize weights and activations in the forward pass while retaining full-precision weights for training. Observers can track value ranges separately from the quantization operation. section 5.2.1 section 5.2.1
