OpenVINO IR is a serialized model artifact stored as an .xml + .bin pair. OpenVINO can compile the IR ahead of time against the CPU plugin’s kernel library; the resulting artifact is ready for loading with ov.Core().compile_model(). section 9.4 section 9.1
Two deployment routes produce different artifacts:
- Standalone: convert ONNX to IR with ov.convert_model, quantize the IR with NNCF, and serve through OpenVINO. section 9.4
- Via ONNX Runtime: keep ORT serving code and load FP32 or QDQ ONNX through the OpenVINO Execution Provider. section 9.4
Listing 9.5 The OpenVINO build pipeline, shared between models
import openvino as ov
import nncf
def build_ov_int8(fp32_onnx, fp32_ir_path, int8_ir_path,
calib_feeds, model_type, ignored_op_names):
fp32_model = ov.convert_model(str(fp32_onnx))
ov.save_model(fp32_model, str(fp32_ir_path),
compress_to_fp16=False) #A
core = ov.Core()
fp32_ir = core.read_model(str(fp32_ir_path))
quantized = nncf.quantize( #B
fp32_ir,
calibration_dataset=nncf.Dataset(calib_feeds, lambda x: x),
preset=nncf.QuantizationPreset.MIXED,
target_device=nncf.TargetDevice.CPU,
model_type=(nncf.ModelType.TRANSFORMER
if model_type == "transformer" else None), #C
ignored_scope=nncf.IgnoredScope(names=ignored_op_names,
validate=False),
fast_bias_correction=True,
)
ov.save_model(quantized, str(int8_ir_path), compress_to_fp16=False)
Inspect the shared build pipeline for converting ONNX, quantizing with NNCF, and producing an OpenVINO model. section 9.4
