liveBook · Manning

OpenVINOExecutionProvider

As explained in Quantization and Fast Inference

The OpenVINO execution provider is an ONNX Runtime backend that delegates model operations to OpenVINO.

OpenVINOExecutionProvider lets ONNX Runtime load FP32 or QDQ ONNX models while delegating execution to the OpenVINO CPU plugin. This route preserves ONNX Runtime serving code, which can simplify maintenance across mixed CPU and GPU fleets. The execution artifact matters: the ORT route and standalone OpenVINO route can perform differently, so benchmark both for the model and deployment. section 9.4

Model-dependent performance

On Sapphire Rapids, via-ORT INT8 reached 96.28 ms for BERT-base, versus 110.17 ms for standalone OpenVINO INT8. For ResNet-18, standalone INT8 reached 41.79 ms, while via-ORT INT8 was 38% slower. These results favor measurement over a blanket route recommendation. section 9.4

How Quantization and Fast Inference covers it

How deep it goes
Explains the ONNX model route and production tradeoffs, but does not detail provider configuration.
Examples
Compares INT8 latency for ResNet-18 and BERT-base on an Intel Sapphire Rapids CPU.
What you can do after
Apply OpenVINO through ONNX Runtime while retaining existing serving code, then benchmark against the standalone route.
How it connects
Connects ONNX Runtime with OpenVINO, ONNX, and INT8 quantization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.