liveBook · Manning

Standalone OpenVINO deployment

As explained in Quantization and Fast Inference

Standalone OpenVINO deployment converts a model to IR, quantizes it with NNCF, and serves it through OpenVINO.

For an Intel-only deployment, convert ONNX to OpenVINO IR, quantize the IR with NNCF, then compile and serve it through OpenVINO. The serialized IR artifact—not the runtime alone—shapes execution. Compare this route with ONNX Runtime’s OpenVINO Execution Provider: which is faster depends on the model. section 9.4

The standalone build proceeds as follows:

  • Benchmark both deployment routes on the target hardware; model-specific results do not support a universal speed ranking. section 9.4

Model-dependent results

On Sapphire Rapids, ResNet-18 native INT8 reached 41.79 ms at batch 32. For BERT-base SST-2, via-ORT INT8 reached 96.28 ms, faster than native INT8 at 110.17 ms. section 9.4

Listing 9.5 The OpenVINO build pipeline, shared between models
import openvino as ov
import nncf
 
def build_ov_int8(fp32_onnx, fp32_ir_path, int8_ir_path,
                  calib_feeds, model_type, ignored_op_names):
    fp32_model = ov.convert_model(str(fp32_onnx))
    ov.save_model(fp32_model, str(fp32_ir_path),
                  compress_to_fp16=False)                                  #A
 
    core = ov.Core()
    fp32_ir = core.read_model(str(fp32_ir_path))
 
    quantized = nncf.quantize(                                             #B
        fp32_ir,
        calibration_dataset=nncf.Dataset(calib_feeds, lambda x: x),
        preset=nncf.QuantizationPreset.MIXED,
        target_device=nncf.TargetDevice.CPU,
        model_type=(nncf.ModelType.TRANSFORMER
                    if model_type == "transformer" else None),             #C
        ignored_scope=nncf.IgnoredScope(names=ignored_op_names,
                                        validate=False),
        fast_bias_correction=True,
    )
    ov.save_model(quantized, str(int8_ir_path), compress_to_fp16=False)
 

Inspect the shared Python pipeline for converting, quantizing, and saving OpenVINO IR. section 9.4

How Quantization and Fast Inference covers it

How deep it goes
Details ONNX conversion, IR quantization, and compilation, then compares measured route performance across models.
Examples
Compares ResNet-18 INT8 latency with BERT-base SST-2 results on Sapphire Rapids.
What you can do after
Build from scratch by converting ONNX, quantizing IR with NNCF, and serving through OpenVINO.
How it connects
Connects ONNX, NNCF quantization, OpenVINO IR, and the OpenVINO CPU plugin.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.