The OpenVINO CPU plugin compiles and executes model graphs on Intel CPUs. Ahead-of-time graph compilation uses the CPU kernel library; for INT8, Q/DQ pairs can be fused into supernodes targeting AMX tile registers. The resulting IR artifact is ready for ov.Core().compile_model(). section 9.4
Choose between two deployment paths: convert ONNX to IR and quantize with NNCF, or keep ONNX Runtime serving code and load FP32 or QDQ ONNX through the OpenVINOExecutionProvider. Both use the plugin, but provide different artifacts; performance depends on the model, so benchmark both paths. section 9.4
Benchmark the model, not just the runtime
On Sapphire Rapids, standalone INT8 ResNet-18 reaches 41.79 ms. For BERT-base SST-2, via-ORT INT8 reaches 96.28 ms, versus 110.17 ms for native INT8. These results favor different paths, making measurement more reliable than a general rule. section 9.4
Listing 9.5 The OpenVINO build pipeline, shared between models
import openvino as ov
import nncf
def build_ov_int8(fp32_onnx, fp32_ir_path, int8_ir_path,
calib_feeds, model_type, ignored_op_names):
fp32_model = ov.convert_model(str(fp32_onnx))
ov.save_model(fp32_model, str(fp32_ir_path),
compress_to_fp16=False) #A
core = ov.Core()
fp32_ir = core.read_model(str(fp32_ir_path))
quantized = nncf.quantize( #B
fp32_ir,
calibration_dataset=nncf.Dataset(calib_feeds, lambda x: x),
preset=nncf.QuantizationPreset.MIXED,
target_device=nncf.TargetDevice.CPU,
model_type=(nncf.ModelType.TRANSFORMER
if model_type == "transformer" else None), #C
ignored_scope=nncf.IgnoredScope(names=ignored_op_names,
validate=False),
fast_bias_correction=True,
)
ov.save_model(quantized, str(int8_ir_path), compress_to_fp16=False)
Inspect the shared build pipeline: it converts ONNX to OpenVINO IR, then applies NNCF quantization targeting the CPU. section 9.4
