Quantization-aware training
Quantization-aware training optimizes a model while simulating or incorporating quantization effects so it can retain performance with reduced-precision representations.
Quantization and Fast Inference
Quantization-aware training incorporates simulated quantization effects while optimizing a model so it can retain accuracy at deployment precision.
- How it is used
- Recovers accuracy when post-training quantization misses deployment targets, and supports training models under ternary constraints.
- How deep it goes
- Works through fake quantization and surrogate gradients, then specifies layer selection, fine-tuning, observer scheduling, and validation practices.
- Examples
- Compares INT4 strategies on FashionMNIST and BERT SST-2, and ternary training against post-hoc quantization on toy regression.
- What you can do after
- Build fake-quantized layers and fine-tuning loops, then evaluate selective precision choices against accuracy and runtime requirements.
Read how Quantization and Fast Inference explains quantization-aware training →
Transformers in Action
Quantization-aware training retrains a model while simulating or applying quantized weights and activations so the model adapts to quantization effects.
- How it is used
- Improves model retention of performance when deploying extremely low-precision weights and activations.
- How deep it goes
- Defines retraining with quantized weights and activations, notes computational cost, and stops short of implementation procedure.
- Examples
- Uses 2-bit quantization as a low-precision target and compares float32, float16, int8, and int4 formats.
- What you can do after
- Evaluate the training-cost and precision trade-off between quantization-aware and posttraining quantization.
Read how Transformers in Action explains quantization-aware training →
Also covered in
What to know first
- bfloat16bfloat16 is a 16-bit floating-point format with one sign bit, eight exponent bits, and seven fraction bits, providing a wide numerical range with reduced precision.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- int4 quantizationINT4 quantization represents numerical values using 4-bit integers, reducing storage requirements while limiting numerical precision.
Related concepts
- quantizationuses · Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
- post-training quantizationalternative to · Post-training quantization converts a trained model’s parameters or computations to lower-precision representations without further model training.
- fake quantizationuses · Fake quantization simulates rounding and clipping to a low-precision representation during the forward computation while retaining differentiable floating-point values for training.
- straight-through estimatoruses · A straight-through estimator approximates the backward derivative of a non-differentiable operation by passing gradients through it according to a surrogate rule.
- forward passuses · A forward pass is a computation that propagates inputs through a model or dependency structure in the forward direction to calculate outputs or intermediate values.
Questions about quantization-aware training
What is quantization-aware training?
Quantization-aware training optimizes a model while simulating or incorporating quantization effects so it can retain performance with reduced-precision representations.
Which Manning books explain quantization-aware training?
Quantization and Fast Inference and Transformers in Action.
What should I know before learning quantization-aware training?
Manning books that teach quantization-aware training usually introduce bfloat16, accuracy, language model and int4 quantization first.
What is the difference between quantization-aware training and post-training quantization?
Quantization-aware training optimizes a model while simulating or incorporating quantization effects so it can retain performance with reduced-precision representations. Post-training quantization converts a trained model’s parameters or computations to lower-precision representations without further model training.
What is quantization-aware training also called?
Quantization-aware training is also called QAT.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

