For an Intel-only deployment, convert ONNX to OpenVINO IR, quantize the IR with NNCF, then compile and serve it through OpenVINO. The serialized IR artifact—not the runtime alone—shapes execution. Compare this route with ONNX Runtime’s OpenVINO Execution Provider: which is faster depends on the model. section 9.4
The standalone build proceeds as follows:
- Benchmark both deployment routes on the target hardware; model-specific results do not support a universal speed ranking. section 9.4
Model-dependent results
On Sapphire Rapids, ResNet-18 native INT8 reached 41.79 ms at batch 32. For BERT-base SST-2, via-ORT INT8 reached 96.28 ms, faster than native INT8 at 110.17 ms. section 9.4
Listing 9.5 The OpenVINO build pipeline, shared between models
import openvino as ov
import nncf
def build_ov_int8(fp32_onnx, fp32_ir_path, int8_ir_path,
calib_feeds, model_type, ignored_op_names):
fp32_model = ov.convert_model(str(fp32_onnx))
ov.save_model(fp32_model, str(fp32_ir_path),
compress_to_fp16=False) #A
core = ov.Core()
fp32_ir = core.read_model(str(fp32_ir_path))
quantized = nncf.quantize( #B
fp32_ir,
calibration_dataset=nncf.Dataset(calib_feeds, lambda x: x),
preset=nncf.QuantizationPreset.MIXED,
target_device=nncf.TargetDevice.CPU,
model_type=(nncf.ModelType.TRANSFORMER
if model_type == "transformer" else None), #C
ignored_scope=nncf.IgnoredScope(names=ignored_op_names,
validate=False),
fast_bias_correction=True,
)
ov.save_model(quantized, str(int8_ir_path), compress_to_fp16=False)
Inspect the shared Python pipeline for converting, quantizing, and saving OpenVINO IR. section 9.4
