4 Applying Post-Training Quantization and Calibration
This chapter covers
- Deciding when post-training quantization is sufficient for your deployment
- Building production-faithful calibration sets
- Range estimation algorithms and their trade-offs
- Applying validation protocols
Chapter 3 established what to quantize—weights, activations, KV cache—and at what granularity—per-tensor, per-channel, or group-wise. You left with a decision checklist that tells you which tensors deserve attention and how finely to slice the scale factors.
But knowing what to quantize is not the same as knowing what data to derive your mapping from. The scale factor S and zero-point Z that define your quantization grid are determined by the activations the model encounters — and those activations depend on the data, not the model. Derive them from calibration data that doesn't match production, and your carefully chosen granularity scheme produces garbage in deployment. For example, calibrate a support chatbot on Wikipedia text, then deploy it on customer tickets full of order IDs and code. Those inputs push activations past the calibrated range, the values get clipped, and answers degrade on the traffic that matters most. Derive them from data that does, and a model quantized in minutes can match original accuracy.