liveBook · Manning

QOperator

As explained in Quantization and Fast Inference

QOperator is an ONNX quantization format that replaces targeted operators with explicit integer operators.

QOperator is an ONNX quantization format that replaces targeted operators with explicit integer operators; for example, MatMul becomes QLinearMatMul. Its graph declaration affects which kernels an execution provider can dispatch. section 9.2

Choose the format with the execution provider in mind:

  • On CPU EP, the BERT benchmark was slower with QOperator than QDQ: 292.7 ms versus 223.4 ms. ORT flags S8S8-QOperator as potentially slow on x86-64. section 9.2
  • On CUDA EP, the same benchmark measured 13.8 ms with QOperator and 14.4 ms with QDQ. Limited INT8 kernel coverage and reference-path fallbacks can make an INT8 artifact a poor choice for CUDA deployment. section 9.2

How Quantization and Fast Inference covers it

How deep it goes
Defines operator replacement and compares provider-specific latency, while noting kernel coverage and fusion flexibility.
Examples
Compares QOperator and QDQ latency for BERT on CPU EP and CUDA EP.
What you can do after
Evaluate QOperator against QDQ using latency measurements for the target execution provider.
How it connects
Contrasts QDQ and connects ONNX quantization to execution providers and integer operator kernels.

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.