Static quantization
Static quantization determines quantization parameters from representative data before inference and keeps them fixed during inference.
CUDA for LLMs
Static quantization computes activation quantization parameters before inference using calibration data and reuses them for subsequent inputs.
- How it is used
- Uses precomputed activation parameters to avoid calibration reductions during inference.
- How deep it goes
- Details calibration, parameter storage, granularity, validation, and profiling, with guidance for choosing alternatives when accuracy suffers.
- Examples
- Uses a customer-support language model and a transformer with different quantization granularities across layers.
- What you can do after
- Evaluate and configure a static quantization workflow using representative calibration data, separate accuracy tests, and profiling.
Quantization and Fast Inference
Static quantization determines quantization ranges during calibration before deployment and keeps the resulting parameters fixed at runtime.
- How it is used
- Sets activation quantization ranges before deployment to avoid computing them during inference.
- How deep it goes
- Explains calibration, observer-based range collection, fixed scales, and accuracy risks under distribution shift.
- Examples
- Uses BERT activation percentiles and a static-versus-dynamic MSE comparison across typical, narrow, wide, and outlier-heavy inputs.
- What you can do after
- Configure it by calibrating representative inputs, freezing scales, and checking quantization scope and runtime precision on the deployment host.
Read how Quantization and Fast Inference explains static quantization →
What to know first
- mean squared errorMean squared error is the arithmetic mean of the squared differences between corresponding predicted and reference numerical values.
Related concepts
- dynamic quantizationalternative to · Dynamic quantization is a method that determines or adjusts quantization parameters at runtime from the values being processed, often quantizing activations during inference.
- quantizationkind of · Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
- quantization calibrationuses · Quantization calibration uses representative data to determine ranges and parameters for mapping model values to a lower-precision representation.
- calibration datauses · Calibration data is data used to estimate or adjust a model’s or system’s parameters, outputs, or behavior for a specified calibration objective.
- QDQ formatuses · QDQ format represents quantized ONNX computations with explicit QuantizeLinear and DequantizeLinear operations around tensors or operators.
Questions about static quantization
What is static quantization?
Static quantization determines quantization parameters from representative data before inference and keeps them fixed during inference.
Which Manning books explain static quantization?
CUDA for LLMs and Quantization and Fast Inference.
What should I know before learning static quantization?
Manning books that teach static quantization usually introduce mean squared error first.
What is the difference between static quantization and dynamic quantization?
Static quantization determines quantization parameters from representative data before inference and keeps them fixed during inference. Dynamic quantization is a method that determines or adjusts quantization parameters at runtime from the values being processed, often quantizing activations during inference.
What is static quantization a kind of?
Static quantization is a kind of quantization. Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

