liveBook · Manning

Chapter 7: Quantizing Large Language Models in Practice

What you will learn in chapter 7 of Quantization and Fast Inference

The chapter introduction explains why LLM quantization must account for unusually large activation values and the memory demands of the KV cache.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduction explains why LLM quantization must account for unusually large activation values and the memory demands of the KV cache.

Key terms: hidden dimensions, outliers, clipping, perplexity, KV cache, long context, concurrent users, tokens-per-second

7.1 Handle outliers and key-value cache considerations

This section explains why large language models need special treatment for persistent, extreme activation outliers and for the growing memory cost of the key-value cache.

Key terms: outlier channels, hidden dimensions, Jaccard similarity, per-tensor INT8, per-channel quantization, scale, residual error, percentile clipping, perplexity, key-value cache

7.2 Apply outlier-aware weight paths

This section explains how LLM.int8() separates activation outliers from an INT8 matrix multiplication, why its threshold matters, and how the approach performs in OPT-6.7B.

Key terms: activation outliers, per-channel weight quantization, per-tensor activation quantization, scale-poisoning, LLM.int8(), outlier feature dimensions, mixed-precision decomposition, per-token activation quantization, threshold 6.0, outlier dimensions

7.3 Apply Hessian-aware groupwise methods (GPTQ)

This section explains how GPTQ uses calibration-derived sensitivity and error compensation to quantize LLM weights, then evaluates its quality, storage trade-offs, and deployment limits.

Key terms: output-space error, Hessian, calibration activations, Hessian diagonal, column-wise quantization, error compensation, inverse Hessian, RTN, fixed column order, lazy batch updates

7.4 Apply activation-aware weight protection (AWQ)

This section explains how AWQ uses activation-guided scaling to protect important weight channels during four-bit quantization, then evaluates its benefits and limits.

Key terms: activation magnitude, weight channels, salient channels, output error, per-channel scaling, equivalent transformation, inverse scaling, LayerNorm absorption, RTN, scaling exponent α

7.5 Compress KV cache with vector quantization (TurboQuant)

This section explains how TurboQuant compresses KV-cache vectors without calibration and evaluates its memory, accuracy, and variant-selection tradeoffs.

Key terms: KV cache, data-oblivious, weight quantization, random orthogonal matrix, rotation, Beta distribution, outlier channels, L2 norm, FP16, normalized direction

7.6 From method comparison to deployment choice

This section compares LLM quantization methods and shows how to choose and combine them according to the deployment constraint that matters most.

Key terms: activation outliers, outlier channels, perplexity, perplexity deltas, evaluation variance, FP16, binding deployment constraint, LLM.int8(), AWQ, GPTQ

7.7 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 6: Porting Workflows Across Frameworks Chapter 8: Pushing into Low Bits Safely →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.