Dynamic quantization
Dynamic quantization is a method that determines or adjusts quantization parameters at runtime from the values being processed, often quantizing activations during inference.
Quantization and Fast Inference
Dynamic range quantization stores model weights at reduced precision and handles activation ranges dynamically during inference.
- How it is used
- Adapts activation ranges to varying inference inputs and supports calibration-free weight compression.
- How deep it goes
- Derives runtime INT8 scaling and per-token quantization, then examines converted tensor dtypes and deployment tradeoffs.
- Examples
- Compares synthetic input distributions and token ranges, BERT SST-2 conversion, and an EfficientNet-Lite0 dtype map.
- What you can do after
- Apply it via TFLite conversion and inspect tensor dtypes to check whether the resulting graph uses floating-point activations.
Read how Quantization and Fast Inference explains dynamic quantization →
CUDA for LLMs
Dynamic quantization computes at least some quantization parameters at runtime from the values being processed.
- How it is used
- Adapts activation scales during inference when input distributions vary.
- How deep it goes
- Explains the timing tradeoff and shows a CUDA absolute-maximum reduction for the first dynamic quantization pass.
- Examples
- Shows a CUDA kernel that reduces input magnitudes to one stored absolute maximum.
- What you can do after
- Evaluate static and dynamic quantization by measuring worst-case accuracy and profiling runtime overhead.
What to know first
- activationAn activation is a numerical value or tensor produced by a neural-network unit, layer, or operation during a forward computation.
- activation rangeAn activation range is the interval of numerical values taken by neural-network activations across inputs.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- dequantizationDequantization maps values from a quantized numerical representation to a higher-precision representation, typically using quantization parameters.
Related concepts
- static quantizationalternative to · Static quantization determines quantization parameters from representative data before inference and keeps them fixed during inference.
- matrix-matrix multiplicationuses · Matrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- quantizationkind of · Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
- reduction operationuses · A reduction operation combines multiple input values according to a rule to produce fewer results, often a single aggregate.
- dequantizationuses · Dequantization maps values from a quantized numerical representation to a higher-precision representation, typically using quantization parameters.
Questions about dynamic quantization
What is dynamic quantization?
Dynamic quantization is a method that determines or adjusts quantization parameters at runtime from the values being processed, often quantizing activations during inference.
Which Manning books explain dynamic quantization?
Quantization and Fast Inference and CUDA for LLMs.
What should I know before learning dynamic quantization?
Manning books that teach dynamic quantization usually introduce activation, activation range, accuracy and dequantization first.
What is the difference between dynamic quantization and static quantization?
Dynamic quantization is a method that determines or adjusts quantization parameters at runtime from the values being processed, often quantizing activations during inference. Static quantization determines quantization parameters from representative data before inference and keeps them fixed during inference.
What is dynamic quantization a kind of?
Dynamic quantization is a kind of quantization. Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

