liveBook · Manning

QDQ format

As explained in Quantization and Fast Inference

QDQ format is an ONNX quantization representation that places QuantizeLinear and DequantizeLinear nodes around operators.

QDQ format represents quantized operators in ONNX graphs with QuantizeLinear and DequantizeLinear nodes. A runtime can fuse these nodes with an operator into a native INT8 kernel. section 6.5

ResNet-18 static quantization

The graph wraps Conv, Add, MaxPool, and GlobalAveragePool operators in QDQ pairs. It reaches 100% operator coverage; runtime fusion reduces 48 compute ops to 31, with 108 QDQ nodes handling data-type transitions. section 6.5

Figure 6.8 Operator quantization coverage by ONNX variant. Left: ResNet-18 ORT Dynamic — 4% coverage, only 1 MatMulInteger node (the classification head's Gemm) out of 51 compute ops. All 20 Conv and 17 Relu stay in FP32. Center: ResNet-18 ORT Static — 100% coverage, all 20 Conv, 8 Add, and the Gemm wrapped in QDQ pairs. Right: BERT-base ORT Dynamic — 28% coverage, 74 MatMulInteger nodes (the linear projections) quantized, but LayerNormalization, Softmax, Div, and Sqrt remain in FP32.

Compare QDQ-based static coverage with dynamic quantization coverage for ResNet-18 and BERT-base. section 6.5

How Quantization and Fast Inference covers it

How deep it goes
Defines QDQ nodes and explains runtime fusion into INT8 kernels, with a static-quantization graph example.
Examples
Uses ResNet-18 static quantization, where QDQ pairs wrap Conv, Add, MaxPool, and GlobalAveragePool operators.
What you can do after
Diagnose operator coverage by identifying operators whose inputs come from DequantizeLinear nodes.
How it connects
Connects ONNX quantization to QuantizeLinear, DequantizeLinear, and INT8 kernels.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.