Chapter 5: Recovering Accuracy with Quantization-Aware Training
The introduction explains when post-training quantization may need help and how quantization-aware training lets a model adapt to low precision.
After this chapter you will be able to
- Limits of Post-Training Quantization. Explain why PTQ is usually the first choice for INT8 but may fall short at INT4, for small models with little redundancy, or under tight accuracy budgets. Describe why QAT should be reserved for cases where a simpler intervention is insufficient.
- Adaptation Through Fake Quantization. Describe how QAT exposes a model to quantization during training so its weights can adapt to reduce loss in the quantized regime.
- Subtle PTQ failure. Explain why PTQ failure is not limited to a dramatic accuracy collapse: degradation may compound across tasks, or edge-case failures may escape the validation set.
- Finding the PTQ accuracy cliff. Explain how the section locates an accuracy cliff by comparing quantized accuracy with an FP32 baseline across bit-widths, and name the deployment pressures that can make PTQ inadequate.
- Remaining signal after INT4. Describe why ResNet-18's INT4 result is seriously degraded yet still a promising regime for intervention.
- Architecture versus model size. Distinguish BERT's INT4 collapse from ResNet-18's degradation, and explain why parameter count alone does not predict which model survives quantization better.
- Layer sensitivity patterns. Describe how to test layer sensitivity and explain what concentrated versus diffuse INT4 degradation implies for mixed precision and QAT.
- PTQ intervention decision. Apply the section's decision framework to choose PTQ, recalibration, mixed precision, or QAT based on accuracy requirements, bit-width, and failure pattern.
- Rounding and vanished gradients. Explain why using the true derivative of rounding in quantization prevents weights from updating during training.
- Fake quantization and STE. Describe how fake quantization and the straight-through estimator give training a quantized forward pass and a usable backward pass. Explain why that backward gradient is an approximation rather than the true derivative.
- Clipped STE gradient boundary. Explain why the clipped STE passes gradients for in-range values but zeros gradients for values clipped to the representable range.
- Fake-Quantized Layer Design. Describe how a fake-quantized linear or convolutional layer uses quantized values in its forward pass while retaining full-precision weights for training.
- Weight Migration Toward Grid. Explain the observed change in weight distributions during the INT4 QAT example and why it helps quantized deployment.
- Observer and Quantizer Separation. Trace how an observer supplies ranges to a fake quantizer and distinguish the two observer strategies described.
- Fake Quantization and Gradient Flow. Explain how fake quantization simulates deployment during the forward pass while the straight-through estimator (STE) allows gradients through rounding, and why the clipped STE blocks gradients for saturated values.
- QAT Fine-Tuning Choices. Describe the training choices in the QAT loop and the reasons given for gradient clipping and cosine annealing.
- Efficient QAT scheduling. Explain why a short fine-tuning run with a gradual transition to quantization can be more effective than longer QAT without a clear schedule.
- QAT Learning Rate Schedule. Explain why QAT uses a learning-rate warmup followed by cosine decay rather than an excessively large or small rate.
- Observer Freezing Handoff. Describe how updating observer statistics moves the quantization grid, and explain when and why to freeze observers.
- Progressive Quantization Schedule. Explain why QAT can begin at a higher bit width and step down to INT4 instead of introducing INT4 quantization immediately.
- Batch Normalization Folding Order. Explain why batch normalization must be folded before inserting fake quantization modules for QAT.
- Three-phase QAT schedule. Trace how the schedule moves from INT8 calibration through an INT6 transition to INT4 target precision, coordinating learning rate and observer state with each phase. Describe how the training loop advances those components.
- Scheduled QAT training outcomes. Explain what the SmallCNN run shows at the INT6 and INT4 transitions, and distinguish the measured accuracy benefit from the stability benefit of scheduled QAT over direct QAT.
- Per-channel gradient survival. Explain why per-channel fake quantization lets more gradients survive during QAT than per-tensor fake quantization.
- Per-Tensor Gradient Loss. Explain how a global INT4 scale can leave small-weight channels poorly represented and how the STE's clipping mask can prevent their weights from learning. Describe what the ResNet-18 conv1 comparison shows.
- Per-Channel Fake Quantization Scales. Describe how per-channel fake quantization computes and broadcasts scales for conv or linear weights, and explain how those scales are handled during QAT.
- Per-channel QAT training impact. Explain why per-channel INT4 QAT showed no accuracy advantage over per-tensor INT4 QAT on SmallCNN, despite the earlier gradient-survival result on pretrained ResNet-18.
- Weight and activation defaults. Justify per-channel quantization as the default for weights, while distinguishing the section's recommendation for activations.
- Output-channel quantization axis. Explain why the section recommends axis 0 for convolutional and linear weights, and identify the depthwise-convolution exception.
- Transformer sublayer sensitivity diagnostic. Describe how to profile transformer sublayers before QAT and explain why FFN weights show more quantization error than attention projection weights in the BERT example.
- Selective INT4 QAT recovery. Explain what the attention-only, FFN-only, and all-layer QAT comparisons show about recovering BERT accuracy from INT4 PTQ.
What it builds on
- Know when post-training quantization is enough (chapter 4) · PTQ as a practical first choice.
- Define the affine mapping with scale and zero point (chapter 2) · The mapping between model values and quantized values.
- Validate accuracy and size with a repeatable protocol (chapter 4) · Validating quantized-model accuracy.
- Identify where precision loss matters (chapter 3) · Assumes that precision loss can affect parts of a model differently.
- Estimate ranges (absolute max, percentile, and MSE) (chapter 4) · Assumes calibration affects PTQ accuracy.
- Characterize granular and overload errors (chapter 2) · The distinction between values within a quantization range and values beyond it.
- Quantize weights with per-tensor and per-channel choices (chapter 3) · Per-channel choices for weights.
- Quantize activations under outliers and dynamic ranges (chapter 3) · Earlier choices for quantizing activations
Section by section
Introduction
The introduction explains when post-training quantization may need help and how quantization-aware training lets a model adapt to low precision.
- Limits of Post-Training Quantization: Explain why PTQ is usually the first choice for INT8 but may fall short at INT4, for small models with little redundancy, or under tight accuracy budgets. Describe why QAT should be reserved for cases where a simpler intervention is insufficient.
Why it matters: It helps you avoid retraining when PTQ already meets the accuracy target. - Adaptation Through Fake Quantization: Describe how QAT exposes a model to quantization during training so its weights can adapt to reduce loss in the quantized regime.
Why it matters: It explains how training can recover accuracy that PTQ alone cannot preserve.
Key terms: post-training quantization, PTQ, INT8, INT4, observe and freeze, quantization-aware training, QAT, fake quantization operators, forward pass, gradients
5.1 Recognize when post-training methods fall short
This section explains that PTQ failure can be subtle, setting up the decision of when QAT may be warranted.
- Subtle PTQ failure: Explain why PTQ failure is not limited to a dramatic accuracy collapse: degradation may compound across tasks, or edge-case failures may escape the validation set.
Why it matters: Looking only for a large accuracy drop could cause you to miss a PTQ failure.
Key terms: PTQ, QAT, accuracy collapse, validation set
5.1.1 The accuracy cliff
This section shows how to spot a post-training quantization accuracy cliff and distinguish a damaged but useful model from one that has lost its learned signal.
- Finding the PTQ accuracy cliff: Explain how the section locates an accuracy cliff by comparing quantized accuracy with an FP32 baseline across bit-widths, and name the deployment pressures that can make PTQ inadequate.
Why it matters: The relevant cliff is the one that crosses your deployment's accuracy limit. - Remaining signal after INT4: Describe why ResNet-18's INT4 result is seriously degraded yet still a promising regime for intervention.
Why it matters: A large accuracy drop does not necessarily mean the quantized model has lost all useful behavior. - Architecture versus model size: Distinguish BERT's INT4 collapse from ResNet-18's degradation, and explain why parameter count alone does not predict which model survives quantization better.
Why it matters: An accuracy number and a model's size can both hide how badly quantization has affected its predictions.
Key terms: PTQ, accuracy cliff, bit-width, FP32 baseline, accuracy delta, INT4, random chance, remaining signal, QAT, BERT-base
5.1.2 Diagnostic signatures of PTQ failure
This section uses layer, per-class, and confidence diagnostics to distinguish PTQ failure patterns and choose between calibration, mixed precision, and QAT.
- Layer sensitivity patterns: Describe how to test layer sensitivity and explain what concentrated versus diffuse INT4 degradation implies for mixed precision and QAT.
Why it matters: The location of the damage determines whether a targeted bit-width change is enough or whole-model retraining is warranted. - Per-class failure signatures: Explain what per-class accuracy reveals about the INT4 failures of ResNet-18 and BERT that overall accuracy alone can hide.
Why it matters: An aggregate accuracy score can conceal a model that has stopped recognizing an entire class. - Confidence distribution shifts: Describe how comparing FP32 and INT4 prediction confidence exposes fragility in ResNet-18 and corroborates BERT's failure.
Why it matters: Confidence shifts can flag fragile predictions even when top-1 accuracy still appears reasonable. - PTQ intervention decision: Apply the section's decision framework to choose PTQ, recalibration, mixed precision, or QAT based on accuracy requirements, bit-width, and failure pattern.
Why it matters: Matching the intervention to the failure avoids unnecessary retraining without overlooking losses that a targeted fix cannot address.
Key terms: layer sensitivity analysis, one-at-a-time perturbation analysis, mixed precision, diffuse failure pattern, QAT, per-class accuracy degradation, mode-collapsed model, aggregate accuracy, confidence distribution shift, maximum softmax probability
5.2 Train with fake quantization and straight-through estimators
This section explains why rounding blocks backpropagation and how fake quantization with a straight-through estimator lets training proceed while handling clipped values.
- Rounding and vanished gradients: Explain why using the true derivative of rounding in quantization prevents weights from updating during training.
Why it matters: This is the obstacle QAT must overcome to train a model that experiences quantization. - Fake quantization and STE: Describe how fake quantization and the straight-through estimator give training a quantized forward pass and a usable backward pass. Explain why that backward gradient is an approximation rather than the true derivative.
Why it matters: It explains how a model can adapt its full-precision weights to the quantized values it sees during training. - Clipped STE gradient boundary: Explain why the clipped STE passes gradients for in-range values but zeros gradients for values clipped to the representable range.
Why it matters: The boundary rule keeps the STE from encouraging updates that drive saturated values farther beyond the quantizable range.
Key terms: round(), zero gradient, backpropagation, fake quantization, straight-through estimator, STE, forward pass, backward pass, clipped STE, qmin
5.2.1 Build a complete fake-quantized layer
This section shows how fake-quantized layers simulate low-bit inference during training, what QAT does to weight distributions, and how observers, STE choices, and stability safeguards support that training.
- Fake-Quantized Layer Design: Describe how a fake-quantized linear or convolutional layer uses quantized values in its forward pass while retaining full-precision weights for training.
Why it matters: This is how QAT exposes a trainable model to the quantization it will face at deployment. - Weight Migration Toward Grid: Explain the observed change in weight distributions during the INT4 QAT example and why it helps quantized deployment.
Why it matters: The snapshots make visible how training adapts weights to the precision they will have at deployment. - Observer and Quantizer Separation: Trace how an observer supplies ranges to a fake quantizer and distinguish the two observer strategies described.
Why it matters: Separating range measurement from quantization makes the fake quantizer adaptable without rewriting its mapping logic. - Clipped STE Choice: Distinguish basic and clipped STE gradients and state the section's recommendation on other variants.
Why it matters: The gradient rule determines which values receive an update through fake quantization. - Fake Quantization Stability Safeguards: Describe the three safeguards against silent numerical failures in fake quantization and the problem each addresses.
Why it matters: These inexpensive checks prevent fake quantization from silently destabilizing training.
Key terms: FakeQuantizedLinear, FakeQuantizedConv2d, per-channel, symmetric, per-tensor, asymmetric, FP32, QAT, INT4 quantization grid, quantization error, weight distributions, observer
5.2.2 The QAT training loop
This section brings fake quantization and straight-through gradients into a QAT training loop and explains the fine-tuning choices that help it remain stable.
- Fake Quantization and Gradient Flow: Explain how fake quantization simulates deployment during the forward pass while the straight-through estimator (STE) allows gradients through rounding, and why the clipped STE blocks gradients for saturated values.
Why it matters: This explains how a model can train in the presence of quantization effects rather than encounter them only after training. - QAT Fine-Tuning Choices: Describe the training choices in the QAT loop and the reasons given for gradient clipping and cosine annealing.
Why it matters: These choices adapt an existing model to quantization while controlling unstable gradients and helping weights settle on the quantization grid.
Key terms: fake quantization, STE, clipped variant, quantization grid, QAT, gradient clipping, cosine annealing, learning rate
5.3 Schedule training to minimize cost
This section explains why QAT needs a cost-conscious schedule rather than simply more training epochs.
- Efficient QAT scheduling: Explain why a short fine-tuning run with a gradual transition to quantization can be more effective than longer QAT without a clear schedule.
Why it matters: A planned transition can avoid spending extra training time without a corresponding benefit. - Sources of QAT training cost: Describe what happens in QAT forward and backward passes and why unnecessary epochs, rather than the per-step penalty, drive its real cost.
Why it matters: This cost breakdown explains why avoiding unnecessary QAT epochs matters.
Key terms: QAT, fine tuning, gradual transition to quantization, fake quantization, STE, memory-bandwidth-bound, epochs
5.3.1 Warm up the learning rate
This subsection explains how learning-rate scheduling, observer freezing, progressive quantization, and batch normalization folding help QAT adapt without creating an inference-time mismatch.
- QAT Learning Rate Schedule: Explain why QAT uses a learning-rate warmup followed by cosine decay rather than an excessively large or small rate.
Why it matters: The schedule helps weights adapt to quantization without destabilizing early training or wasting later epochs. - Observer Freezing Handoff: Describe how updating observer statistics moves the quantization grid, and explain when and why to freeze observers.
Why it matters: A stable grid stops training from chasing changing quantization targets. - Progressive Quantization Schedule: Explain why QAT can begin at a higher bit width and step down to INT4 instead of introducing INT4 quantization immediately.
Why it matters: Introducing quantization noise gradually can avoid the abrupt loss increase caused by a direct jump to INT4. - Batch Normalization Folding Order: Explain why batch normalization must be folded before inserting fake quantization modules for QAT.
Why it matters: Folding first makes QAT account for the weights that will actually be quantized at deployment.
Key terms: warmup, cosine decay, STE, quantization grid, observers, scale factors, freeze, progressive quantization, INT8, INT4
5.3.2 Put the schedule together
This section combines bit-width changes, learning-rate scheduling, and observer freezing into a complete QAT schedule, then evaluates its training behavior against direct QAT.
- Three-phase QAT schedule: Trace how the schedule moves from INT8 calibration through an INT6 transition to INT4 target precision, coordinating learning rate and observer state with each phase. Describe how the training loop advances those components.
Why it matters: Coordinating the changes makes the move to INT4 a training process rather than a single abrupt switch. - Practical scheduling choices: State the section’s guidelines for folding batch normalization, allocating training time across precision phases, warming and decaying the learning rate, and choosing when to freeze observers.
Why it matters: These guidelines turn the example timeline into a schedule a reader can adapt to another QAT run. - Scheduled QAT training outcomes: Explain what the SmallCNN run shows at the INT6 and INT4 transitions, and distinguish the measured accuracy benefit from the stability benefit of scheduled QAT over direct QAT.
Why it matters: The results show why a smoother, more debuggable run can matter even when final accuracy gains are small.
Key terms: INT8, INT6, INT4, observers, quantization grid, cosine decay, batch normalization, max relative scale change, progressive quantization, cosine annealing
5.4 Use per-channel strategies for convolution and linear layers
This section explains how the scale granularity of a fake quantizer affects gradient survival during quantization-aware training.
- Per-channel gradient survival: Explain why per-channel fake quantization lets more gradients survive during QAT than per-tensor fake quantization.
Why it matters: Scale choice affects not only quantization error but also whether weights can keep adapting during training.
Key terms: per-tensor fake quantization, per-channel quantization, straight-through estimator, clipping boundaries, scale
5.4.1 The gradient survival problem
Explains why per-tensor INT4 fake quantization can suppress gradients in channels with small weights and how per-channel scales preserve their ability to learn during QAT.
- Per-Tensor Gradient Loss: Explain how a global INT4 scale can leave small-weight channels poorly represented and how the STE's clipping mask can prevent their weights from learning. Describe what the ResNet-18 conv1 comparison shows.
Why it matters: A channel with no surviving gradients cannot adapt during quantization-aware training. - Per-Channel Fake Quantization Scales: Describe how per-channel fake quantization computes and broadcasts scales for conv or linear weights, and explain how those scales are handled during QAT.
Why it matters: Channel-specific scales let small-weight channels retain their internal structure without making scale computation part of the gradient path.
Key terms: per-tensor quantization, INT4, STE, clipped, gradient survival, per-channel fake quantization, per-channel scales, q_max, broadcasting, STE mask
5.4.2 Measure the training impact
This section measures when per-channel QAT improves training, explains why the benefit varies by model, and gives practical choices for scales and quantization axes.
- Per-channel QAT training impact: Explain why per-channel INT4 QAT showed no accuracy advantage over per-tensor INT4 QAT on SmallCNN, despite the earlier gradient-survival result on pretrained ResNet-18.
Why it matters: A large gradient-survival improvement on one model does not guarantee an accuracy gain on another. - Weight and activation defaults: Justify per-channel quantization as the default for weights, while distinguishing the section's recommendation for activations.
Why it matters: The default retains a potential accuracy benefit without adding training compute, but it should not be applied indiscriminately to activations. - Per-channel scale stability: Describe how per-channel abs-max scales behaved during SmallCNN training and what the section says about pretrained-model fine-tuning.
Why it matters: Recomputing scales every forward pass did not itself destabilize the SmallCNN experiment. - Output-channel quantization axis: Explain why the section recommends axis 0 for convolutional and linear weights, and identify the depthwise-convolution exception.
Why it matters: Choosing the output axis gives distinct feature detectors their own scales rather than grouping them under input-channel scales.
Key terms: per-channel QAT, per-tensor QAT, gradient survival, STE masks, per-channel quantization, per-tensor, per-token, activations, abs-max calibration, per-channel scales
5.5 Adapt transformers efficiently
This section uses sublayer sensitivity profiling to choose transformer QAT strategies, shows how selective INT4 QAT recovers accuracy, and identifies components and limits that require different precision choices.
- Transformer sublayer sensitivity diagnostic: Describe how to profile transformer sublayers before QAT and explain why FFN weights show more quantization error than attention projection weights in the BERT example.
Why it matters: A short diagnostic can identify where low-precision weights are likely to cost the most accuracy. - Selective INT4 QAT recovery: Explain what the attention-only, FFN-only, and all-layer QAT comparisons show about recovering BERT accuracy from INT4 PTQ.
Why it matters: The experiment shows both that QAT can rescue a failed INT4 model and that layer choice still affects accuracy. - Precision choices by sublayer: Apply the section's INT8 and INT4 guidance to transformer linear layers, including when to retain higher precision for FFN weights.
Why it matters: The bit width and measured sensitivity determine whether a linear sublayer is a good candidate for lower precision. - FP32 transformer holdouts: Justify keeping LayerNorm, softmax, and GELU in FP32 rather than including them in the weight-QAT strategies.
Why it matters: Leaving these operations in FP32 avoids risking stability for little weight-compression benefit. - Weight-only QAT scale limit: Explain why the BERT weight-QAT strategy is not sufficient on its own for the larger LLMs described here.
Why it matters: A successful weight-QAT result on BERT should not be mistaken for a complete solution to LLM quantization.
Key terms: sublayer diagnostic, output MSE, FP32 baseline, FFN, attention projections, weight quantization, INT4 PTQ, QAT, STE, attention-only
5.6 Summary
Where this is used later
- Verify equivalence and diagnose coverage (chapter 6) · Deployment-equivalence checks need to catch mismatches between trained quantization parameters and exported weights.
- Handle outliers and key-value cache considerations (chapter 7) · Distinguishing weight sensitivity from activation outliers helps frame LLM quantization choices.
- From method comparison to deployment choice (chapter 7) · Comparing quantization choices requires weighing quality costs rather than assuming all layers behave alike.
Concepts explained in this chapter
- quantization-aware trainingQuantization-aware training incorporates simulated quantization effects while optimizing a model so it can retain accuracy at deployment precision.
- fake quantizationFakeQuantize is a module that applies simulated quantization and dequantization using configured quantization parameters during computation.
- clipped straight-through estimatorA clipped straight-through estimator passes gradients for values within a specified range and sets gradients outside that range to zero.
- straight-through estimatorA straight-through estimator approximates gradients through a nondifferentiable operation using a surrogate derivative.
- per-channel quantizationPer-channel quantization assigns separate quantization parameters to each tensor channel.
- QAT decision frameworkThe QAT decision framework is a diagnostic process for selecting an intervention based on quantization failure mode and bit-width.
- per-class accuracy analysisPer-class accuracy analysis evaluates a classifier's accuracy separately for each target class.
- gradientA gradient is a vector of derivatives describing how a function changes with respect to its inputs or parameters.
- progressive quantizationProgressive quantization introduces successively lower numerical precision over stages rather than applying the final precision immediately.
- warmupLearning-rate warmup gradually increases the learning rate from a smaller initial value during an early phase of training.
- quantization errorGranular error is the bounded quantization error caused by rounding a value to a nearby grid point.
- quantization gridA quantization grid is a discrete set of representable numerical levels with a defined spacing.
- feedforward networkA feed-forward network is a neural-network component that transforms representations through successive learned layers without recurrent connections.
- batch normalization foldingBatch normalization folding is a graph transformation that incorporates batch-normalization parameters into an adjacent operation such as a convolution.
- layer sensitivity analysisLayer sensitivity analysis measures how perturbing or quantizing individual layers affects model behavior or accuracy.
- post-training quantizationPost-training quantization converts a trained model to lower-precision representations after training, typically using calibration data.
- granularityThe level of tensor partitioning that determines which values share quantization parameters such as a scale factor.
- cosine annealingCosine annealing is a schedule that smoothly decreases the learning rate along a cosine-shaped curve.
- absmax quantizationA quantization range-estimation method that sets boundaries using the largest observed absolute value.
- per-tensor quantizationPer-tensor quantization uses one set of quantization parameters for all values in a tensor.
- per-token quantizationPer-token quantization assigns quantization parameters separately to the values associated with each input token.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
