liveBook · Manning

OpenVINO IR

As explained in Quantization and Fast Inference

OpenVINO IR is an intermediate model representation stored in XML and binary files for use with OpenVINO tools and runtimes.

OpenVINO IR is a serialized model artifact stored as an .xml + .bin pair. OpenVINO can compile the IR ahead of time against the CPU plugin’s kernel library; the resulting artifact is ready for loading with ov.Core().compile_model(). section 9.4 section 9.1

Two deployment routes produce different artifacts:

  • Standalone: convert ONNX to IR with ov.convert_model, quantize the IR with NNCF, and serve through OpenVINO. section 9.4
  • Via ONNX Runtime: keep ORT serving code and load FP32 or QDQ ONNX through the OpenVINO Execution Provider. section 9.4
Listing 9.5 The OpenVINO build pipeline, shared between models
import openvino as ov
import nncf
 
def build_ov_int8(fp32_onnx, fp32_ir_path, int8_ir_path,
                  calib_feeds, model_type, ignored_op_names):
    fp32_model = ov.convert_model(str(fp32_onnx))
    ov.save_model(fp32_model, str(fp32_ir_path),
                  compress_to_fp16=False)                                  #A
 
    core = ov.Core()
    fp32_ir = core.read_model(str(fp32_ir_path))
 
    quantized = nncf.quantize(                                             #B
        fp32_ir,
        calibration_dataset=nncf.Dataset(calib_feeds, lambda x: x),
        preset=nncf.QuantizationPreset.MIXED,
        target_device=nncf.TargetDevice.CPU,
        model_type=(nncf.ModelType.TRANSFORMER
                    if model_type == "transformer" else None),             #C
        ignored_scope=nncf.IgnoredScope(names=ignored_op_names,
                                        validate=False),
        fast_bias_correction=True,
    )
    ov.save_model(quantized, str(int8_ir_path), compress_to_fp16=False)
 

Inspect the shared build pipeline for converting ONNX, quantizing with NNCF, and producing an OpenVINO model. section 9.4

How Quantization and Fast Inference covers it

How deep it goes
Explains conversion, NNCF quantization, and CPU-plugin compilation, but does not detail the IR file schema.
Examples
Compares standalone and via-ORT latency for ResNet-18 and BERT-base on Sapphire Rapids CPUs.
What you can do after
Build an INT8 IR pipeline by converting ONNX, quantizing with NNCF, and compiling through ov.Core().compile_model().
How it connects
Connects IR to ONNX conversion, NNCF quantization, OpenVINO, and ONNX Runtime deployment routes.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.