5 Recovering Accuracy with Quantization-Aware Training
This chapter covers
- Recognizing when post-training quantization falls short
- Training with fake quantization and straight-through estimators
- Scheduling QAT to minimize training cost
- Per-channel strategies for convolution and linear layers
- Adapting transformers efficiently
Chapter 4 established post-training quantization as the practitioner's first choice—fast, cheap, and effective for the majority of INT8 deployments. For ResNet-18 and BERT-base at INT8, PTQ delivered near-lossless compression with zero retraining.
But PTQ has limits. When you push to INT4, when you quantize small models with little redundancy, when your accuracy budget is measured in tenths of a percent—PTQ's "observe and freeze" approach runs out of runway. The model has learned nothing about operating in low precision; it simply endures whatever distortion quantization imposes.
Quantization-aware training (QAT) changes the contract. Instead of hoping the model tolerates quantization, we teach it to expect quantization. We insert fake quantization operators into the forward pass during training, let gradients flow through them, and allow the model to adapt its weights to minimize loss in the quantized regime. The weights learn to position themselves favorably on the integer grid, avoiding regions where rounding error accumulates.