liveBook · Manning

Operator coverage analysis

As explained in Quantization and Fast Inference

Operator coverage analysis identifies which operators in a model graph execute in a specified numerical format.

Operator coverage analysis counts which graph operators execute in INT8 versus FP32. It helps explain why quantization may compress a model without speeding inference: operators left in FP32 do not benefit from integer execution. section 6.5

Interpret coverage by checking which operators the quantization format marks as quantized:

  • In QDQ-format ONNX graphs, inputs from DequantizeLinear nodes identify operators that can have the QDQ pattern fused into an INT8 kernel. section 6.5
  • In operator-replacement graphs, distinct nodes such as MatMulInteger or DynamicQuantizeLinear indicate quantized operators. section 6.5

Coverage explains different outcomes

ResNet-18 ORT Dynamic quantizes 4% of compute operations: its MatMul/Gemm classification head is quantized, while its convolution layers remain FP32. ORT Static reaches 100% coverage. BERT-base ORT Dynamic reaches 28%, leaving operators such as LayerNormalization and Softmax in FP32.

How Quantization and Fast Inference covers it

How deep it goes
Explains how QDQ and replacement-node formats reveal quantized operators, then compares coverage across model variants.
Examples
Compares ResNet-18 ORT Dynamic at 4% coverage, ResNet-18 ORT Static at 100%, and BERT-base ORT Dynamic at 28%.
What you can do after
Diagnose why quantization may not speed inference by identifying operators that remain in FP32.
How it connects
Connects quantization coverage to INT8 inference, ONNX graph formats, and inference speedup.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.