Quantization calibration
Quantization calibration uses representative data to determine ranges and parameters for mapping model values to a lower-precision representation.
Quantization and Fast Inference
Calibration is the process of measuring representative model values to determine their actual numerical range for configuring quantization.
- How it is used
- Uses representative activation data to configure post-training quantization for deployment without retraining.
- How deep it goes
- Derives symmetric and affine parameters, implements a running min-max observer, and contrasts clipping with rounding error.
- Examples
- Uses an activation observer, an outlier-driven scale comparison, and a quantized linear layer compared with FP32.
- What you can do after
- Build it from scratch by collecting activation ranges and deriving symmetric or affine quantization parameters.
Read how Quantization and Fast Inference explains quantization calibration →
CUDA for LLMs
Quantization calibration determines quantization parameters from representative data to map values into a lower-precision representation.
- How it is used
- Determines reusable quantization parameters from representative data for static inference.
- How deep it goes
- Explains range collection, min-max and percentile selection, granularity, validation, and runtime tradeoffs.
- Examples
- Uses an activation outlier, transformer attention heads, and 4096 × 4096 feed-forward weights to illustrate range and granularity choices.
- What you can do after
- Evaluate calibration choices using separate validation data, downstream task metrics, and performance profiling.
Also covered in
- Rearchitecting LLMsIn early access (MEAP)
What to know first
- activationAn activation is a numerical value or tensor produced by a neural-network unit, layer, or operation during a forward computation.
- attention headAn attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- dequantizationDequantization maps values from a quantized numerical representation to a higher-precision representation, typically using quantization parameters.
Related concepts
- min-max calibrationincludes · Min-max calibration determines quantization parameters from the minimum and maximum values observed in representative data.
- percentile-based calibrationincludes · Percentile-based calibration sets quantization ranges from selected percentiles of observed values, potentially clipping values outside those ranges.
- quantization scaleproduces · A quantization scale is a multiplicative factor relating quantized values or steps to the numerical values they represent.
- static quantizationused by · Static quantization determines quantization parameters from representative data before inference and keeps them fixed during inference.
Questions about quantization calibration
What is quantization calibration?
Quantization calibration uses representative data to determine ranges and parameters for mapping model values to a lower-precision representation.
Which Manning books explain quantization calibration?
Quantization and Fast Inference and CUDA for LLMs.
What should I know before learning quantization calibration?
Manning books that teach quantization calibration usually introduce activation, attention head, accuracy and dequantization first.
What kinds of quantization calibration are there?
Kinds of quantization calibration include min-max calibration and percentile-based calibration.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

