Chapter 3: Choosing What to Quantize and at What Granularity
This introduction explains why weights, activations, and the KV cache need different quantization choices and why scale granularity must balance accuracy with practical constraints.
After this chapter you will be able to
- Distinct Quantization Targets. Explain how weights, activations, and the transformer KV cache differ at inference, and why quantizing them all identically can waste precision or mishandle error-sensitive values.
- Scale Granularity Trade-Off. Describe the trade-off among a single scale per tensor, a scale per output channel, and group-wise scales, and state what guides the choice.
- Weight Distributions and Quantization. Explain why frozen weights can be analyzed offline and why a roughly zero-centered distribution suits symmetric quantization, while skewed distributions may need asymmetric quantization.
- Dynamic Activation Ranges. Explain why activation ranges need representative calibration and how varying inputs and outliers create a clipping-versus-resolution trade-off.
- Activation Quantization Compute Benefit. Distinguish the primarily memory-saving effect of weight-only quantization from the integer-compute benefit possible when both weights and activations are quantized.
- KV Cache Memory Pressure. Trace how a transformer KV cache avoids recomputing earlier tokens' keys and values, why its memory grows with context and batch size, and why quantizing it chiefly targets memory.
- Layer Sensitivity Analysis. Describe how quantizing one layer at a time and measuring accuracy identifies layers that are especially sensitive to precision loss.
- Weight scale granularity. Describe how symmetric INT8 weight quantization computes and applies a scale for an entire tensor versus each output channel.
- Outlier penalty and precision recovery. Explain why a global scale wastes precision on small-magnitude channels and how per-channel scales address that loss.
- Accuracy versus scale metadata. Justify the section’s default choice of per-channel weight quantization using its accuracy benefit and scale-storage cost.
- Output channel scale axis. Apply the output-channel rule to locate per-channel weight scales in linear and convolutional layers.
- Input-Dependent Ranges and Outliers. Explain why a fixed activation scale can lose accuracy across inputs and why protecting rare large values reduces resolution for typical values.
- Static and Dynamic Scales. Distinguish when static and dynamic quantization compute activation scales, and explain why their accuracy differs when inputs depart from calibration data.
- Per-Token Transformer Scales. Explain how per-token dynamic quantization assigns scales in a transformer sequence and why it can reduce error compared with a single dynamic per-tensor scale.
- Structured Transformer Outlier Channels. Describe how persistent outlier channels are identified and why their consistency makes targeted mixed precision possible.
- Structure matched quantization granularity. Justify per-channel scales for keys and per-token scales for values using their observed outliers and the section’s equal-bit-width error comparison.
- Low Bit Granularity Tradeoff. Explain why per-channel scaling can work for INT8 but produce substantial error for INT4, and how smaller groups improve the fit at the cost of more scales.
- Group Scale Organization. Describe how a linear layer's weights are partitioned into input-axis groups with a separate scale, and how the number of scales depends on the layer dimensions and group size.
- Contiguous Group Kernel Access. Explain why grouping along the input dimension of a row-major linear layer supports efficient memory access and dot-product dequantization.
- INT4 Packing And Alignment. Describe why two INT4 weights are packed per byte, how the signed values are placed and recovered, and why group boundaries must align with bytes.
- Quantization Decision Sequence. Explain the order of decisions: choose what to quantize, the scope of each scale factor, and the bit-width, then verify the configuration against the target hardware. Describe how model sensitivity, latency, accuracy tolerance, and runtime support affect those choices.
- Tensor Target Selection. Describe which weights, activations, and transformer KV caches the checklist calls for quantizing, including the stated exceptions and conditions.
- Transformer Granularity Choices. Describe the recommended transformer choices for weights, activations, sensitive layers, and KV-cache keys and values.
- Runtime Compatibility Gate. Explain why the chosen granularity must be checked against the target runtime before the configuration is finalized.
What it builds on
- Define the affine mapping with scale and zero point (chapter 2) · Introduces scale as part of the quantization mapping.
- Compare symmetric and asymmetric choices (chapter 2) · Assumes the distinction between symmetric and asymmetric quantization.
- The cost crisis in memory, latency, and power (chapter 1) · Assumes the cost of moving model data from memory during inference.
- Characterize granular and overload errors (chapter 2) · Assumes the distinction between quantization error within a range and clipping outside it.
- End-to-end INT8 inference in PyTorch (chapter 2) · Assumes the basic INT8 inference path.
- What integers give up, and what they gain (chapter 1) · Assumes the storage advantage of narrower integers.
Section by section
Introduction
This introduction explains why weights, activations, and the KV cache need different quantization choices and why scale granularity must balance accuracy with practical constraints.
- Distinct Quantization Targets: Explain how weights, activations, and the transformer KV cache differ at inference, and why quantizing them all identically can waste precision or mishandle error-sensitive values.
Why it matters: The right precision choice starts with recognizing that these tensors behave differently. - Scale Granularity Trade-Off: Describe the trade-off among a single scale per tensor, a scale per output channel, and group-wise scales, and state what guides the choice.
Why it matters: Choosing what to quantize is only part of the decision; scale granularity also affects the result.
Key terms: weights, activations, KV cache, outliers, per-tensor, per-channel, group-wise, scale
3.1 Identify where precision loss matters
This section compares the precision risks and benefits of quantizing weights, activations, and transformer KV caches, then shows how to identify particularly sensitive layers.
- Weight Distributions and Quantization: Explain why frozen weights can be analyzed offline and why a roughly zero-centered distribution suits symmetric quantization, while skewed distributions may need asymmetric quantization.
Why it matters: The distribution tells you whether a simple symmetric mapping is a good fit for a weight tensor. - Weight Memory Savings and Exceptions: Describe why reducing weight precision directly reduces model storage and memory traffic, and why some small or sensitive weight-related tensors may remain in higher precision.
Why it matters: A smaller weight representation can ease a memory bottleneck without requiring every tensor to use the same precision. - Dynamic Activation Ranges: Explain why activation ranges need representative calibration and how varying inputs and outliers create a clipping-versus-resolution trade-off.
Why it matters: An activation scale chosen from the wrong data can lose useful precision even when the quantizer itself is implemented correctly. - Activation Quantization Compute Benefit: Distinguish the primarily memory-saving effect of weight-only quantization from the integer-compute benefit possible when both weights and activations are quantized.
Why it matters: It explains why the harder activation target matters when the aim is maximum inference speedup. - KV Cache Memory Pressure: Trace how a transformer KV cache avoids recomputing earlier tokens' keys and values, why its memory grows with context and batch size, and why quantizing it chiefly targets memory.
Why it matters: A long-context cache can consume more memory than the weights, yet its quantization errors can affect attention. - Layer Sensitivity Analysis: Describe how quantizing one layer at a time and measuring accuracy identifies layers that are especially sensitive to precision loss.
Why it matters: Measuring layer-by-layer effects helps avoid spending precision where it matters least.
Key terms: weights, symmetric quantization, asymmetric quantization, zero-point, weight quantization, model size, embedding tables, classification heads, bias vectors, activations
3.2 Quantize weights with per-tensor and per-channel choices
This section compares per-tensor and per-channel weight quantization, showing why channel-specific scales usually preserve more precision at little memory cost.
- Weight scale granularity: Describe how symmetric INT8 weight quantization computes and applies a scale for an entire tensor versus each output channel.
Why it matters: The choice of which weights share a scale determines how much of the integer range each channel can use. - Outlier penalty and precision recovery: Explain why a global scale wastes precision on small-magnitude channels and how per-channel scales address that loss.
Why it matters: A small aggregate error can conceal severe information loss in particular output channels. - Accuracy versus scale metadata: Justify the section’s default choice of per-channel weight quantization using its accuracy benefit and scale-storage cost.
Why it matters: A granularity choice should account for both model quality and the bytes needed to store scales. - Transformer weight sensitivity: Explain why the section finds per-tensor weight quantization especially problematic for transformers compared with CNNs.
Why it matters: The same wasted integer range can have different consequences in different architectures. - Output channel scale axis: Apply the output-channel rule to locate per-channel weight scales in linear and convolutional layers.
Why it matters: Selecting the wrong axis would assign scales to something other than the output features the scheme is meant to protect.
Key terms: per-tensor quantization, per-channel quantization, scale, output channel, INT8, outlier penalty, integer range utilization, quantization error, scale metadata, weight-only quantization
3.3 Quantize activations under outliers and dynamic ranges
This section explains why input-dependent ranges and outliers make activation quantization difficult, then compares static, dynamic, per-token, and mixed-precision responses.
- Input-Dependent Ranges and Outliers: Explain why a fixed activation scale can lose accuracy across inputs and why protecting rare large values reduces resolution for typical values.
Why it matters: It explains why choosing one activation range is harder than choosing a range for fixed weights. - Static and Dynamic Scales: Distinguish when static and dynamic quantization compute activation scales, and explain why their accuracy differs when inputs depart from calibration data.
Why it matters: The timing of scale computation determines whether a deployment favors low runtime overhead or adaptation to changing inputs. - Per-Token Transformer Scales: Explain how per-token dynamic quantization assigns scales in a transformer sequence and why it can reduce error compared with a single dynamic per-tensor scale.
Why it matters: Adapting within a sequence preserves precision for quieter tokens even when other tokens have much larger activations. - Structured Transformer Outlier Channels: Describe how persistent outlier channels are identified and why their consistency makes targeted mixed precision possible.
Why it matters: Predictable outlier channels allow precision to be reserved for the small part of a transformer that needs it.
Key terms: activation statistics, range ratio, outliers, scale, quantization levels, static quantization, dynamic quantization, calibration, observer, scale factors
3.4 Quantize transformer key-value cache with mixed precision
This section explains why transformer keys and values benefit from different KV-cache quantization granularities, tests that choice, and shows its scale layout and storage cost.
- KV cache attention roles: Trace how cached keys and values participate in attention, and explain why their different roles suggest different statistical patterns.
Why it matters: The different jobs of keys and values explain why one quantization granularity need not fit both. - Structure matched quantization granularity: Justify per-channel scales for keys and per-token scales for values using their observed outliers and the section’s equal-bit-width error comparison.
Why it matters: Matching scales to the observed outlier pattern reduces reconstruction error without treating keys and values identically. - Asymmetric scale tensor layout: Describe which dimensions the illustrated key and value quantizers reduce to produce their respective scale tensors.
Why it matters: The dimensions reduced in code determine whether the scales actually match the intended key or value structure. - Default choice and scale cost: State the recommended default, its limits, and the approximate storage cost of per-token value scales in the example.
Why it matters: The default is useful only if its accuracy benefit is weighed against scale storage and checked against the target model.
Key terms: KV cache, query, key, value, attention weights, per-channel, per-token, outlier channels, outlier tokens, MSE
3.5 Use group and block schemes and understand memory layout
The section explains how group and block quantization trade scale granularity against accuracy and metadata cost, and why memory layout and INT4 packing matter for efficient inference.
- Low Bit Granularity Tradeoff: Explain why per-channel scaling can work for INT8 but produce substantial error for INT4, and how smaller groups improve the fit at the cost of more scales.
Why it matters: This explains why an INT8 granularity choice may not carry over to INT4. - Group Scale Organization: Describe how a linear layer's weights are partitioned into input-axis groups with a separate scale, and how the number of scales depends on the layer dimensions and group size.
Why it matters: The grouping scheme determines both the stored metadata and how weights are processed. - Contiguous Group Kernel Access: Explain why grouping along the input dimension of a row-major linear layer supports efficient memory access and dot-product dequantization.
Why it matters: An accuracy-friendly grouping must also match the access pattern of the inference kernel. - Two Dimensional Block Scaling: Distinguish the section's 2D block quantization from 1D group quantization and describe how a block receives its scale.
Why it matters: The section uses “block” in a specific sense that differs from some LLM terminology. - INT4 Packing And Alignment: Describe why two INT4 weights are packed per byte, how the signed values are placed and recovered, and why group boundaries must align with bytes.
Why it matters: Packing is necessary to realize INT4's storage savings without awkward group boundaries. - INT4 Group Size Selection: Apply the section's production guidance to choose an INT4 weight group size according to accuracy, compression, and hardware constraints.
Why it matters: A practical group-size choice balances quality and memory savings against what the hardware supports.
Key terms: granularity, per-channel, per-group, INT4, MSE, scale factor, group quantization, group size, input dimension, scale factors
3.6 Follow a decision checklist for granularity
This section turns quantization choices into a checklist for selecting targets, granularity, and bit-width, then checking architecture needs, runtime support, and common failure signs.
- Quantization Decision Sequence: Explain the order of decisions: choose what to quantize, the scope of each scale factor, and the bit-width, then verify the configuration against the target hardware. Describe how model sensitivity, latency, accuracy tolerance, and runtime support affect those choices.
Why it matters: It keeps a quantization choice tied to the model, performance goal, accuracy needs, and deployment hardware. - Tensor Target Selection: Describe which weights, activations, and transformer KV caches the checklist calls for quantizing, including the stated exceptions and conditions.
Why it matters: Choosing targets first distinguishes the usual weight savings from activation speedups and context-dependent cache needs. - CNN And Vision First Layers: Describe the section's CNN weight and calibration guidance, and explain why a CNN's first convolution or a vision transformer's patch embedding may need FP16.
Why it matters: The usual CNN setting need not force image-input layers into a precision that harms their outputs. - Transformer Granularity Choices: Describe the recommended transformer choices for weights, activations, sensitive layers, and KV-cache keys and values.
Why it matters: Different transformer tensors have different sensitivity and ranges, so their precision choices need not match. - Runtime Compatibility Gate: Explain why the chosen granularity must be checked against the target runtime before the configuration is finalized.
Why it matters: A configuration on paper does not guarantee that the deployed runtime uses that quantization scheme. - Failure Signs And Remedies: Apply the section's quick fixes to distinguish broad accuracy loss, missing speedup, corrupted attention, bad first-layer outputs, a single accuracy-killing layer, and slow group quantization.
Why it matters: Matching a failure sign to its likely adjustment avoids changing the entire model blindly.
Key terms: target selection, granularity, scale factor scope, bit-width, hardware compatibility gate, weights, activations, calibration data, outlier channels, KV cache
3.7 Summary
Where this is used later
- Diagnostic signatures of PTQ failure (chapter 5) · Assumes that precision loss can affect parts of a model differently.
- Build a calibration set that actually represents production (chapter 4) · Addresses how to select representative calibration data.
- Validate accuracy and size with a repeatable protocol (chapter 4) · Provides context for validating benefits beyond reduced model size.
- Handle outliers and key-value cache considerations (chapter 7) · Revisits KV cache considerations in LLM quantization.
- Apply outlier-aware weight paths (chapter 7) · Motivates attention to outlier-affected weight quantization.
- Use per-channel strategies for convolution and linear layers (chapter 5) · Supports per-channel strategies for convolution and linear layers.
- Estimate ranges (absolute max, percentile, and MSE) (chapter 4) · Motivates comparing range-estimation methods.
- Know when post-training quantization is enough (chapter 4) · Activation outliers as a quantization difficulty.
Concepts explained in this chapter
- activation quantizationRepresenting neural-network activations with reduced-precision numerical formats.
- kv cache quantizationKV cache quantization is the reduced-precision representation of the attention keys and values stored for prior tokens.
- perturbation testPerturbation analysis evaluates how changing or disturbing one component affects a system's output or measured behavior.
- quantization calibrationCalibration is the process of measuring representative model values to determine their actual numerical range for configuring quantization.
- weight-only quantizationReducing the precision of model weights while leaving activations at higher precision.
- block quantizationA technique that assigns scaling parameters separately to fixed-size blocks of numerical values rather than to an entire tensor.
- per-channel quantizationPer-channel quantization assigns separate quantization parameters to each tensor channel.
- per-tensor quantizationPer-tensor quantization uses one set of quantization parameters for all values in a tensor.
- per-token quantizationPer-token quantization assigns quantization parameters separately to the values associated with each input token.
- dynamic quantizationDynamic range quantization stores model weights at reduced precision and handles activation ranges dynamically during inference.
- sub-byte packingSub-byte packing combines multiple values narrower than one byte into byte-aligned storage units.
- mixed precisionMixed precision uses different numerical precisions for different tensors or operations within a computation.
- outlier channelAn outlier channel or dimension is a feature coordinate whose activation magnitudes are substantially larger than those of most other coordinates.
- static quantizationStatic quantization determines quantization ranges during calibration before deployment and keeps the resulting parameters fixed at runtime.
- granularityThe level of tensor partitioning that determines which values share quantization parameters such as a scale factor.
- bit precisionThe number of bits used to represent each numerical value.
- activationA numerical output produced by a neural-network layer or operation and passed to subsequent computation.
- key-value cacheRuntime storage for attention keys and values from prior tokens, allowing an autoregressive transformer to reuse them when processing later tokens.
- quantizationQuantization is the representation of numerical values such as neural-network weights and activations with reduced-precision formats.
- quantization scaleA scale factor is the multiplicative parameter that maps between quantized or fixed-point integers and their represented numerical values.
- data driftDistribution mismatch is a difference between the value distribution represented by calibration data and that encountered in production.
- token-wise structureToken-wise variation is a pattern in which numerical magnitudes differ across token positions.
- observerAn observer is a software component that records representative tensor statistics during calibration to determine quantization parameters.
- residual connectionA residual connection adds a block's input to the output of a transformation applied to that input.
- accuracy toleranceAccuracy tolerance is the amount of predictive accuracy degradation acceptable for a model or task.
- memory layoutMemory layout is the arrangement of data elements at addresses in memory.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
