liveBook · Manning

OpenVINO CPU plugin

As explained in Quantization and Fast Inference

The OpenVINO CPU plugin is the OpenVINO backend that compiles and executes model graphs on CPUs.

The OpenVINO CPU plugin compiles and executes model graphs on Intel CPUs. Ahead-of-time graph compilation uses the CPU kernel library; for INT8, Q/DQ pairs can be fused into supernodes targeting AMX tile registers. The resulting IR artifact is ready for ov.Core().compile_model(). section 9.4

Choose between two deployment paths: convert ONNX to IR and quantize with NNCF, or keep ONNX Runtime serving code and load FP32 or QDQ ONNX through the OpenVINOExecutionProvider. Both use the plugin, but provide different artifacts; performance depends on the model, so benchmark both paths. section 9.4

Benchmark the model, not just the runtime

On Sapphire Rapids, standalone INT8 ResNet-18 reaches 41.79 ms. For BERT-base SST-2, via-ORT INT8 reaches 96.28 ms, versus 110.17 ms for native INT8. These results favor different paths, making measurement more reliable than a general rule. section 9.4

Listing 9.5 The OpenVINO build pipeline, shared between models
import openvino as ov
import nncf
 
def build_ov_int8(fp32_onnx, fp32_ir_path, int8_ir_path,
                  calib_feeds, model_type, ignored_op_names):
    fp32_model = ov.convert_model(str(fp32_onnx))
    ov.save_model(fp32_model, str(fp32_ir_path),
                  compress_to_fp16=False)                                  #A
 
    core = ov.Core()
    fp32_ir = core.read_model(str(fp32_ir_path))
 
    quantized = nncf.quantize(                                             #B
        fp32_ir,
        calibration_dataset=nncf.Dataset(calib_feeds, lambda x: x),
        preset=nncf.QuantizationPreset.MIXED,
        target_device=nncf.TargetDevice.CPU,
        model_type=(nncf.ModelType.TRANSFORMER
                    if model_type == "transformer" else None),             #C
        ignored_scope=nncf.IgnoredScope(names=ignored_op_names,
                                        validate=False),
        fast_bias_correction=True,
    )
    ov.save_model(quantized, str(int8_ir_path), compress_to_fp16=False)
 

Inspect the shared build pipeline: it converts ONNX to OpenVINO IR, then applies NNCF quantization targeting the CPU. section 9.4

How Quantization and Fast Inference covers it

How deep it goes
Explains ahead-of-time compilation, INT8 fusion, IR artifacts, and route-dependent performance, but not kernel internals.
Examples
Compares standalone and via-ORT INT8 latency for ResNet-18 and BERT-base SST-2 on Sapphire Rapids.
What you can do after
Apply it via OpenVINO or ONNX Runtime's OpenVINOExecutionProvider, then benchmark paths for the target model.
How it connects
Connects ONNX, NNCF quantization, OpenVINO IR, ONNX Runtime, INT8, and AMX.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.