liveBook · Manning

ONNX

As explained in Quantization and Fast Inference

ONNX is an open standard for representing machine-learning models as portable computational graphs across frameworks and runtimes.

ONNX is an open format for portable machine-learning models. It represents each model as a directed acyclic graph of named tensor inputs and outputs, with operators defined by standardized opsets and graph data serialized using Protocol Buffers. ONNX is the format; ONNX Runtime and other engines execute it. section 6.4

Use ONNX to carry a model from its training framework to a different deployment runtime, avoiding a rewrite of the model code. The same graph format can feed ONNX Runtime, TensorRT, OpenVINO, or CoreML converters. section 6.4

ResNet-18 quantization

ONNX Runtime dynamic quantization covers MatMul only and reaches 1.03× compression. Static quantization with calibration covers all operators, reaches 3.95× compression, and loses 0.10 percentage points of accuracy. section 6.4

Listing 6.6 Export ResNet-18 to ONNX
torch.onnx.export(
    model,
    dummy_input,                                       #A
    "resnet18_fp32.onnx",
    opset_version=17,                                  #B
    input_names=["input"],
    output_names=["output"],
    dynamic_axes={"input": {0: "batch"},
                  "output": {0: "batch"}},             #C
    dynamo=False,                                      #D
)

Inspect a PyTorch ResNet-18 export to an ONNX file using opset 17 and a dynamic batch axis. section 6.4

quantize_static(
    model_input=preprocessed_path,
    model_output="resnet18_static_int8.onnx",
    calibration_data_reader=calib_reader,
    quant_format=QuantFormat.QDQ,                      #A
    activation_type=QuantType.QUInt8,                  #B
    weight_type=QuantType.QInt8,                       #C
    per_channel=True,                                  #D
)

Inspect the ONNX Runtime static quantization configuration, including calibration, QDQ format, and per-channel weights. section 6.4

How Quantization and Fast Inference covers it

How deep it goes
Defines directed acyclic tensor graphs, standardized opsets, and Protocol Buffers serialization, while distinguishing the format from execution engines.
Examples
Compares ResNet-18 and BERT quantization, including MatMul-only versus broader operator coverage and resulting compression.
What you can do after
Apply it by exporting a PyTorch ResNet-18 to opset 17, then producing a statically quantized ONNX artifact with ONNX Runtime.
How it connects
Links PyTorch and TensorFlow models with ONNX Runtime, TensorRT, OpenVINO, and CoreML through a shared graph representation.

Also explained in

Compare how 9 books explain ONNX →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.