liveBook · Manning

OpenVINO through ONNX Runtime

As explained in Quantization and Fast Inference

OpenVINO through ONNX Runtime is a deployment pattern that retains ONNX Runtime serving code and uses the OpenVINO execution provider.

Keep ONNX Runtime serving code and load an FP32 or QDQ ONNX model through OpenVINOExecutionProvider. Both this route and standalone OpenVINO use the OpenVINO CPU plugin, but supply it different artifacts; performance depends on the model, so benchmark both routes. section 9.4

Model-specific latency

On Sapphire Rapids, BERT-base INT8 via ORT ran in 96.28 ms versus 110.17 ms for standalone OpenVINO INT8. For ResNet-18, standalone INT8 was faster; via-ORT INT8 cost 38% relative to native. These results favor measurement over a universal route recommendation. section 9.4

How Quantization and Fast Inference covers it

How deep it goes
Compares deployment routes and reports model-specific latency, while leaving QDQ ONNX preparation to a separate workflow.
Examples
Compares INT8 latency for ResNet-18 and BERT-base on Sapphire Rapids.
What it assumes
Assumes ONNX Runtime serving code and an FP32 or QDQ ONNX model; does not establish these prerequisites from zero.
What you can do after
Apply OpenVINOExecutionProvider to existing ONNX Runtime serving code, then benchmark against standalone OpenVINO.
How it connects
Connects ONNX Runtime, OpenVINO, QDQ quantization, and Intel AMX execution.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.