liveBook · Manning

ONNX Runtime

As explained in Quantization and Fast Inference

ONNX Runtime is an inference engine for executing models represented in the Open Neural Network Exchange format.

ONNX is a portable model format; ONNX Runtime (ORT) is an inference engine that executes ONNX files. This separation lets a model trained in one framework run in a different deployment environment. section 6.4

ORT also quantizes ONNX models. Dynamic quantization covers MatMul in the ResNet-18 comparison, while calibrated static quantization reaches convolution layers too: it achieves 3.95× compression with 0.10 percentage-point accuracy loss, versus 1.03× for dynamic quantization. section 6.4

Listing 6.6 Export ResNet-18 to ONNX
torch.onnx.export(
    model,
    dummy_input,                                       #A
    "resnet18_fp32.onnx",
    opset_version=17,                                  #B
    input_names=["input"],
    output_names=["output"],
    dynamic_axes={"input": {0: "batch"},
                  "output": {0: "batch"}},             #C
    dynamo=False,                                      #D
)

Exporting a PyTorch ResNet-18 model with opset 17, named inputs and outputs, and a dynamic batch axis creates an ONNX artifact for runtime use. section 6.4

How Quantization and Fast Inference covers it

How deep it goes
Distinguishes ONNX portability from ORT execution, then demonstrates export and quantization without detailing ORT internals.
Examples
Uses ResNet-18 export and calibration, and compares ResNet-18 and BERT-base quantization results.
What it assumes
Assumes an exported ONNX graph; static quantization additionally requires calibration data.
What you can do after
Apply dynamic or calibrated static quantization to exported ONNX models, weighing operator coverage against compression and accuracy.
How it connects
Connects ONNX Runtime with ONNX, PyTorch, TensorRT, OpenVINO, CoreML, TFLite, and TorchAO.

Also explained in

Compare how 5 books explain ONNX Runtime →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.