Percentile-based calibration
Percentile-based calibration sets quantization ranges from selected percentiles of observed values, potentially clipping values outside those ranges.
Quantization and Fast Inference
Percentile calibration sets a quantization range using selected distribution percentiles, excluding values beyond those boundaries.
- How it is used
- Selects activation ranges for post-training quantization without letting rare extremes dictate the scale.
- How deep it goes
- Explains threshold computation and error tradeoffs, then compares percentile sweeps with alternative range estimators.
- Examples
- Uses BERT layer 6 activation statistics and a percentile sweep to compare clipping, scale, and reconstruction error.
- What you can do after
- Apply it via a PyTorch percentile observer and evaluate thresholds using clipping rates and reconstruction error.
Read how Quantization and Fast Inference explains percentile-based calibration →
CUDA for LLMs
Percentile calibration sets quantization ranges using distribution percentiles, potentially clipping extreme values.
- How it is used
- Uses distribution percentiles to set quantization ranges that are more robust to outliers.
- How deep it goes
- Explains threshold selection and clipping with numeric examples, including absolute-value percentiles for symmetric quantization.
- Examples
- Uses 1,000 values with 990 in the range 0–5 and 10 outliers near 100 to illustrate a 99th-percentile threshold.
- What you can do after
- Apply percentile-based calibration to choose quantization ranges from representative data while limiting outlier influence.
Read how CUDA for LLMs explains percentile-based calibration →
What to know first
- activationAn activation is a numerical value or tensor produced by a neural-network unit, layer, or operation during a forward computation.
- attention headAn attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- mean squared errorMean squared error is the arithmetic mean of the squared differences between corresponding predicted and reference numerical values.
Related concepts
- min-max calibrationalternative to · Min-max calibration determines quantization parameters from the minimum and maximum values observed in representative data.
- quantization calibrationkind of · Quantization calibration uses representative data to determine ranges and parameters for mapping model values to a lower-precision representation.
- clipping-resolution trade-offuses · The clipping-resolution trade-off describes how narrowing a quantization range increases clipping while improving the spacing-based precision of values that remain in range.
- overload erroruses · Overload error is quantization error caused when a value falls outside the representable range and is clipped to a boundary.
- anomalyaddresses · An anomaly is an observation, data state, or behavior that substantially deviates from an expected or prevailing pattern or condition.
- perplexitycontrasts with · Perplexity is a language-model metric, typically the exponential of average cross-entropy, that measures how well a model predicts a sequence, with lower values indicating better predictive fit.
Questions about percentile-based calibration
What is percentile-based calibration?
Percentile-based calibration sets quantization ranges from selected percentiles of observed values, potentially clipping values outside those ranges.
Which Manning books explain percentile-based calibration?
Quantization and Fast Inference and CUDA for LLMs.
What should I know before learning percentile-based calibration?
Manning books that teach percentile-based calibration usually introduce activation, attention head, accuracy and mean squared error first.
What is the difference between percentile-based calibration and min-max calibration?
Percentile-based calibration sets quantization ranges from selected percentiles of observed values, potentially clipping values outside those ranges. Min-max calibration determines quantization parameters from the minimum and maximum values observed in representative data.
What is percentile-based calibration a kind of?
Percentile-based calibration is a kind of quantization calibration. Quantization calibration uses representative data to determine ranges and parameters for mapping model values to a lower-precision representation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

