liveBook · Manning

Quantize_static

As explained in Quantization and Fast Inference

quantize_static is an ONNX Runtime API for statically quantizing an ONNX model.

Use quantize_static to quantize an ONNX model’s weights and activations using representative calibration data. Unlike dynamic quantization, it can cover operators such as Conv layers that weight-only approaches may leave in FP32. section 6.4

The workflow has three steps:

  • Preprocess the FP32 graph with shape inference and graph optimization. section 6.4
  • Calibrate with representative data to collect activation ranges. section 6.4
  • Quantize the graph by inserting QuantizeLinear/DequantizeLinear (QDQ) pairs. section 6.4

ResNet-18

On ImageNette, static quantization produces an 11.28 MB model, a 3.95× reduction, with 0.10 percentage points of accuracy loss. Dynamic quantization reaches only 1.03× because it targets the single Gemm layer rather than the model’s Conv layers. section 6.4

quantize_static(
    model_input=preprocessed_path,
    model_output="resnet18_static_int8.onnx",
    calibration_data_reader=calib_reader,
    quant_format=QuantFormat.QDQ,                      #A
    activation_type=QuantType.QUInt8,                  #B
    weight_type=QuantType.QInt8,                       #C
    per_channel=True,                                  #D
)

The call configures QDQ format, unsigned 8-bit activations, signed 8-bit weights, and per-channel quantization. section 6.4

How Quantization and Fast Inference covers it

How deep it goes
Explains preprocessing, activation calibration, and QDQ insertion, and identifies configuration options for formats and quantized types.
Examples
Compares static and dynamic quantization on ResNet-18, reporting model compression and accuracy loss.
What it assumes
Requires a valid FP32 ONNX model and representative calibration data; the workflow needs no PyTorch, TensorFlow, or GPU.
What you can do after
Apply ONNX Runtime’s quantize_static API to produce a calibrated, statically quantized ONNX model.
How it connects
Connects ONNX Runtime quantization with ONNX export, graph preprocessing, calibration data readers, and dynamic quantization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.