liveBook · Manning

NNCF

As explained in Quantization and Fast Inference

NNCF is OpenVINO's neural network compression framework for optimizing models, including through quantization.

NNCF (Neural Network Compression Framework) quantizes an OpenVINO IR model for CPU deployment. In the standalone workflow, convert ONNX to IR, quantize the IR with NNCF, then compile it with OpenVINO. The resulting artifact—not simply the choice of runtime—shapes execution. section 9.4

The build pipeline uses calibration data, a mixed quantization preset, and a CPU target. Alternatively, keep ONNX Runtime serving code and use the OpenVINO Execution Provider with an FP32 or QDQ ONNX model. Benchmark both routes: on BERT-base, via-ORT INT8 measured 96.28 ms versus 110.17 ms for native INT8. section 9.4

Listing 9.5 The OpenVINO build pipeline, shared between models
import openvino as ov
import nncf
 
def build_ov_int8(fp32_onnx, fp32_ir_path, int8_ir_path,
                  calib_feeds, model_type, ignored_op_names):
    fp32_model = ov.convert_model(str(fp32_onnx))
    ov.save_model(fp32_model, str(fp32_ir_path),
                  compress_to_fp16=False)                                  #A
 
    core = ov.Core()
    fp32_ir = core.read_model(str(fp32_ir_path))
 
    quantized = nncf.quantize(                                             #B
        fp32_ir,
        calibration_dataset=nncf.Dataset(calib_feeds, lambda x: x),
        preset=nncf.QuantizationPreset.MIXED,
        target_device=nncf.TargetDevice.CPU,
        model_type=(nncf.ModelType.TRANSFORMER
                    if model_type == "transformer" else None),             #C
        ignored_scope=nncf.IgnoredScope(names=ignored_op_names,
                                        validate=False),
        fast_bias_correction=True,
    )
    ov.save_model(quantized, str(int8_ir_path), compress_to_fp16=False)
 

Inspect the shared Python build pipeline for converting ONNX, preparing calibration data, and quantizing an OpenVINO IR model with NNCF. section 9.4

How Quantization and Fast Inference covers it

How deep it goes
Shows calibration-based CPU quantization configuration and deployment routes, but does not detail NNCF internals.
Examples
Compares native and via-ORT INT8 latency on BERT-base, measuring 110.17 ms and 96.28 ms respectively.
What you can do after
Apply NNCF to quantize an OpenVINO IR model using calibration data and a CPU target.
How it connects
Connects NNCF quantization with OpenVINO, ONNX Runtime, ONNX, and INT8 inference.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.