liveBook · Manning

OpenVINO runtime model inspector

As explained in Quantization and Fast Inference

The OpenVINO runtime model inspector examines a compiled runtime graph and reports the precision used by its operations.

Use `compiled.get_runtime_model()` to verify which precisions operations actually execute in a compiled OpenVINO graph. Interpret precision fractions within that graph: runtimes that fuse operations differently use different counting denominators. section 9.4

ResNet-18 precision check

On the native INT8 graph, 19 of 89 compute operations run in i8 and 21 in u8, for a 44.9% INT8 fraction. The stem, classifier, and some BatchNorm and residual operations remain in floating point. section 9.4

How Quantization and Fast Inference covers it

How deep it goes
Reports operation-level precision counts, but fractions depend on graph granularity and are not directly comparable across runtimes.
Examples
Checks ResNet-18’s 89 compute operations, including 19 i8 and 21 u8 operations.
What it assumes
Assumes a compiled OpenVINO model that exposes a runtime graph through `compiled.get_runtime_model()`.
What you can do after
Evaluate whether low-precision operations actually execute in the compiled graph.
How it connects
Connects INT8 and BF16 execution with graph fusion and TensorRT engine inspection.

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.