chapter nine

9 Quantization

 

This chapter covers

  • How FP32, INT8, and INT4 formats trade memory for accuracy
  • Implementing quantization/dequantization steps and reasoning about error
  • Choosing granularity schemes that balance hardware cost and fidelity
  • Calibrating scales and zero points with static or dynamic approaches
  • Navigating symmetric versus asymmetric choices alongside AWQ refinements
  • Stitching these ideas into an end-to-end model quantization workflow

Picture yourself trying to run a modern language model on a single GPU. A 70B-parameter model needs about 140 GB for its weights in FP16, and your GPU has 80 GB. At 4 bits per weight, the same model takes about 35 GB and fits on one 48 GB workstation card. The technique that gets you there is quantization.

When you compress a model from 16-bit to 4-bit precision, it needs 4× less memory to load and 4× less bandwidth to move data around, and inference runs twice as fast. Your GPU bill gets cut in half. It’s the difference between “impossible on this hardware” and “runs smoothly in production.”

Most models barely notice. A carefully weight-quantized model often maintains around 99% of its original accuracy on common benchmarks. Think about that tradeoff: 4× smaller, roughly 2× faster in memory-bound decoding, and you lose maybe 1% accuracy. The speed gain is workload dependent (it comes from moving fewer bytes, so bandwidth-bound inference benefits most), but for most real-world applications, this is close to a free lunch.

9.1 Quantization building blocks

9.1.1 Data types: The foundation of quantization

9.1.2 Basic quant/dequant operations

9.2 Deployment strategies

9.2.1 Granularity schemes: How to apply quantization

9.2.2 Calibration: Computing quantization parameters

9.2.3 Static vs. dynamic quantization

9.2.4 Symmetric vs. asymmetric quantization

9.3 Advanced techniques

9.3.1 Activation-aware weight quantization

9.3.2 Simple example: Big vs. small inputs

9.4 Putting it all together

9.4.1 Real model quantization workflow

9.4.2 Accuracy validation strategy

9.4.3 Common pitfalls

9.4.4 The tradeoffs in one place