7 Quantizing Large Language Models in Practice
This chapter covers
- Why LLMs need a new quantization toolkit: activation outliers and KV-cache pressure
- LLM.int8(): routing outlier dimensions to FP16
- GPTQ: Hessian-guided error compensation for 4-bit weights
- AWQ: activation-aware scaling to protect salient channels
- TurboQuant for the KV cache, and choosing a method for your deployment
Quantization so far has meant squeezing a well-behaved tensor into fewer bits. Large language models break that assumption. A handful of hidden dimensions inside a 7B or 70B model carry magnitudes hundreds of times larger than their neighbors, and those dimensions are load-bearing — clip them and the model's perplexity collapses by an order of magnitude.
Any practitioner shipping an LLM runs into this the first time they try to serve one on commodity GPUs. Memory, concurrent users, and tokens-per-second all hinge on how you handle these outliers, and on a second bottleneck that only appears at serving time: the KV cache, which at long context can equal the model itself in size.
Four production methods — LLM.int8(), GPTQ, AWQ, and TurboQuant — each solve a piece of the puzzle. We measure where the standard toolkit breaks, build each method up, and finish with a deployment decision.