liveBook · Manning

Operator coverage

As explained in Quantization and Fast Inference

The proportion or extent of a model's operators affected by an optimization or quantization method.

Operator coverage describes how much of a model is affected by an optimization or quantization method. Check it alongside latency: an INT8 label does not guarantee that every operator executes in INT8, and low operator-count coverage can still be effective when the covered operators dominate compute. section 6.5 section 6.6

Interpret a coverage percentage by its counting unit. An ONNX graph can show which operators were quantized or wrapped in quantize/dequantize (QDQ) pairs. A TensorRT engine inspector instead reports the execution precision of compiled compute layers; fusion can change the layer count. For the reported engine fractions, Reformat, Constant, and Shuffle layers are excluded. These percentages are not necessarily directly comparable. section 6.5 section 9.3

Figure 6.8 Operator quantization coverage by ONNX variant. Left: ResNet-18 ORT Dynamic — 4% coverage, only 1 MatMulInteger node (the classification head's Gemm) out of 51 compute ops. All 20 Conv and 17 Relu stay in FP32. Center: ResNet-18 ORT Static — 100% coverage, all 20 Conv, 8 Add, and the Gemm wrapped in QDQ pairs. Right: BERT-base ORT Dynamic — 28% coverage, 74 MatMulInteger nodes (the linear projections) quantized, but LayerNormalization, Softmax, Div, and Sqrt remain in FP32.

Compare graph-level coverage across dynamic and static ONNX Runtime quantization, including operators left in FP32. section 6.5

Coverage depends on the graph and quantization path

For ResNet-18, ONNX Runtime dynamic quantization reports 4% coverage: the classification head becomes a MatMulInteger node, while all 20 Conv and 17 Relu nodes remain FP32. Static QDQ quantization reports 100% coverage, with all 20 Conv, eight Add, and the Gemm wrapped in QDQ pairs. Dynamic quantization is more effective on BERT, whose linear projections provide more MatMul targets. section 6.5 section 6.6

Inspect the compiled engine

On an NVIDIA L4, TensorRT implicit INT8 reaches a 95.7% INT8 fraction and 1.04 ms latency for ResNet-18, but 0.0% and 3.77 ms for BERT-base. Explicit INT8 reaches 83.3% and 1.73 ms for ResNet-18, versus 97.8% and 2.73 ms for BERT. Fusion and precision placement affect both the counts and latency, so measure the built engine rather than inferring execution from the build flag. section 9.3

Table 9.2 Engine sizes and execution-precision distribution, L4. Compute layers exclude Reformat/Constant/Shuffle.
Model
Engine
Compute layers
INT8 fraction
Size (MB)
Latency (ms)
ResNet-18
FP32
25
0.0%
47.0
6.66
ResNet-18
FP16
23
0.0%
23.8
2.67
ResNet-18
INT8 implicit
23
95.7%
12.1
1.04
ResNet-18
INT8 explicit
24
83.3%
13.9
1.73
BERT-base
FP32
124
0.0%
438.6
8.77
BERT-base
FP16
86
0.0%
221.1
3.79
BERT-base
INT8 implicit
86
0.0%
221.1
3.77
BERT-base
INT8 explicit
183
97.8%
184.8
2.73

Inspect engine sizes, compute-layer counts, INT8 fractions, and latency together for both models. section 9.3

Coverage also does not establish numerical equivalence. Run identical inputs through the FP32 baseline and each variant, compare raw logits and prediction agreement, and check the FP32 export first to separate export error from quantization error. Then inspect coverage and benchmark the deployed runtime. section 6.5 section 6.6

How Quantization and Fast Inference covers it

How deep it goes
Compares graph-level coverage and engine execution fractions, then relates their counts to latency and precision placement.
Examples
Compares ResNet-18 and BERT-base under ONNX Runtime quantization and TensorRT implicit and explicit INT8 engines.
What you can do after
Diagnoses precision coverage gaps by inspecting quantized graphs and compiled engine execution fractions alongside latency.
How it connects
Connects operator coverage to quantization, numerical equivalence, operator fusion, and inference latency.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.