liveBook · Manning

Dynamic quantization

Dynamic quantization is a method that determines or adjusts quantization parameters at runtime from the values being processed, often quantizing activations during inference.

How 2 Manning books explain it

Quantization and Fast Inference

Dynamic range quantization stores model weights at reduced precision and handles activation ranges dynamically during inference.

How it is used
Adapts activation ranges to varying inference inputs and supports calibration-free weight compression.
How deep it goes
Derives runtime INT8 scaling and per-token quantization, then examines converted tensor dtypes and deployment tradeoffs.
Examples
Compares synthetic input distributions and token ranges, BERT SST-2 conversion, and an EfficientNet-Lite0 dtype map.
What you can do after
Apply it via TFLite conversion and inspect tensor dtypes to check whether the resulting graph uses floating-point activations.

Read how Quantization and Fast Inference explains dynamic quantization →

CUDA for LLMs

Dynamic quantization computes at least some quantization parameters at runtime from the values being processed.

How it is used
Adapts activation scales during inference when input distributions vary.
How deep it goes
Explains the timing tradeoff and shows a CUDA absolute-maximum reduction for the first dynamic quantization pass.
Examples
Shows a CUDA kernel that reduces input magnitudes to one stored absolute maximum.
What you can do after
Evaluate static and dynamic quantization by measuring worst-case accuracy and profiling runtime overhead.

Read how CUDA for LLMs explains dynamic quantization →

What to know first

Related concepts

Questions about dynamic quantization

What is dynamic quantization?

Dynamic quantization is a method that determines or adjusts quantization parameters at runtime from the values being processed, often quantizing activations during inference.

Which Manning books explain dynamic quantization?

Quantization and Fast Inference and CUDA for LLMs.

What should I know before learning dynamic quantization?

Manning books that teach dynamic quantization usually introduce activation, activation range, accuracy and dequantization first.

What is the difference between dynamic quantization and static quantization?

Dynamic quantization is a method that determines or adjusts quantization parameters at runtime from the values being processed, often quantizing activations during inference. Static quantization determines quantization parameters from representative data before inference and keeps them fixed during inference.

What is dynamic quantization a kind of?

Dynamic quantization is a kind of quantization. Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.