liveBook · Manning

Percentile-based calibration

Percentile-based calibration sets quantization ranges from selected percentiles of observed values, potentially clipping values outside those ranges.

How 2 Manning books explain it

Quantization and Fast Inference

Percentile calibration sets a quantization range using selected distribution percentiles, excluding values beyond those boundaries.

How it is used
Selects activation ranges for post-training quantization without letting rare extremes dictate the scale.
How deep it goes
Explains threshold computation and error tradeoffs, then compares percentile sweeps with alternative range estimators.
Examples
Uses BERT layer 6 activation statistics and a percentile sweep to compare clipping, scale, and reconstruction error.
What you can do after
Apply it via a PyTorch percentile observer and evaluate thresholds using clipping rates and reconstruction error.

Read how Quantization and Fast Inference explains percentile-based calibration →

CUDA for LLMs

Percentile calibration sets quantization ranges using distribution percentiles, potentially clipping extreme values.

How it is used
Uses distribution percentiles to set quantization ranges that are more robust to outliers.
How deep it goes
Explains threshold selection and clipping with numeric examples, including absolute-value percentiles for symmetric quantization.
Examples
Uses 1,000 values with 990 in the range 0–5 and 10 outliers near 100 to illustrate a 99th-percentile threshold.
What you can do after
Apply percentile-based calibration to choose quantization ranges from representative data while limiting outlier influence.

Read how CUDA for LLMs explains percentile-based calibration →

What to know first

Related concepts

Questions about percentile-based calibration

What is percentile-based calibration?

Percentile-based calibration sets quantization ranges from selected percentiles of observed values, potentially clipping values outside those ranges.

Which Manning books explain percentile-based calibration?

Quantization and Fast Inference and CUDA for LLMs.

What should I know before learning percentile-based calibration?

Manning books that teach percentile-based calibration usually introduce activation, attention head, accuracy and mean squared error first.

What is the difference between percentile-based calibration and min-max calibration?

Percentile-based calibration sets quantization ranges from selected percentiles of observed values, potentially clipping values outside those ranges. Min-max calibration determines quantization parameters from the minimum and maximum values observed in representative data.

What is percentile-based calibration a kind of?

Percentile-based calibration is a kind of quantization calibration. Quantization calibration uses representative data to determine ranges and parameters for mapping model values to a lower-precision representation.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.