liveBook · Manning

ONNX dynamic quantization

As explained in Quantization and Fast Inference

quantize_dynamic is an ONNX Runtime API for dynamically quantizing an ONNX model.

ONNX Runtime’s `quantize_dynamic` applies weight-only quantization to an ONNX graph, scaling activations on the fly. It works directly on valid FP32 ONNX files, without PyTorch, TensorFlow, or a GPU. section 6.4

By default, it targets MatMul and Gemm operators, so its impact depends on the model’s operator mix. On ResNet-18, that means quantizing only the classifier Gemm: the model shrinks from 43.12 MB by just 1.03×. For this convolution-heavy model, static quantization is the more effective path. section 6.4

How Quantization and Fast Inference covers it

How deep it goes
Defines on-the-fly activation scaling and default MatMul/Gemm targets, but does not detail the quantization internals.
Examples
Compares ResNet-18’s 1.03× reduction with BERT-base’s 3.97× compression under ONNX Runtime dynamic quantization.
What it assumes
Requires a valid FP32 ONNX file; the workflow does not require PyTorch or TensorFlow.
What you can do after
Apply ONNX Runtime’s `quantize_dynamic` to a valid FP32 ONNX graph without requiring a GPU.
How it connects
Connects ONNX Runtime quantization with ONNX graphs, MatMul, Gemm, and static quantization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.