NNCF (Neural Network Compression Framework) quantizes an OpenVINO IR model for CPU deployment. In the standalone workflow, convert ONNX to IR, quantize the IR with NNCF, then compile it with OpenVINO. The resulting artifact—not simply the choice of runtime—shapes execution. section 9.4
The build pipeline uses calibration data, a mixed quantization preset, and a CPU target. Alternatively, keep ONNX Runtime serving code and use the OpenVINO Execution Provider with an FP32 or QDQ ONNX model. Benchmark both routes: on BERT-base, via-ORT INT8 measured 96.28 ms versus 110.17 ms for native INT8. section 9.4
Listing 9.5 The OpenVINO build pipeline, shared between models
import openvino as ov
import nncf
def build_ov_int8(fp32_onnx, fp32_ir_path, int8_ir_path,
calib_feeds, model_type, ignored_op_names):
fp32_model = ov.convert_model(str(fp32_onnx))
ov.save_model(fp32_model, str(fp32_ir_path),
compress_to_fp16=False) #A
core = ov.Core()
fp32_ir = core.read_model(str(fp32_ir_path))
quantized = nncf.quantize( #B
fp32_ir,
calibration_dataset=nncf.Dataset(calib_feeds, lambda x: x),
preset=nncf.QuantizationPreset.MIXED,
target_device=nncf.TargetDevice.CPU,
model_type=(nncf.ModelType.TRANSFORMER
if model_type == "transformer" else None), #C
ignored_scope=nncf.IgnoredScope(names=ignored_op_names,
validate=False),
fast_bias_correction=True,
)
ov.save_model(quantized, str(int8_ir_path), compress_to_fp16=False)
Inspect the shared Python build pipeline for converting ONNX, preparing calibration data, and quantizing an OpenVINO IR model with NNCF. section 9.4
