liveBook · Manning

Explicit quantization

As explained in Quantization and Fast Inference

Explicit quantization is a quantization mode in which quantization locations and parameters are represented in the model graph and honored by the inference-engine builder.

Explicit quantization pins quantization placements in ONNX with Q/DQ nodes, which TensorRT must honor when building an INT8 engine. It is the recommended TensorRT path, but it does not guarantee the lowest latency for every model. section 9.3

Prepare the ONNX artifact for TensorRT by:

  • Using QDQ format rather than QOperator. section 9.3
  • Setting activations to symmetric quantization. section 9.3
  • Restricting quantized operators to Conv or MatMul, and excluding quantization-sensitive layers near inputs and outputs. section 9.3

Latency depends on the model

On an L4, explicit INT8 runs ResNet-18 at 1.73 ms, slower than implicit INT8 at 1.04 ms; BERT-base runs at 2.73 ms, while implicit INT8 is near FP16 latency. section 9.3

Figure 9.7 BERT-base SST-2 latency on L4, batch 8, seq 128. Explicit INT8 leads at 2.73 ms; implicit INT8 lands within measurement noise of FP16. Dotted lines mark Table 9.1's CUDA EP INT8 (14.4 ms) and FP32-optimized (9 ms) references. TRT's INT8 explicit closes the gap section 9.2 identified.

Compare the BERT-base latency results across TensorRT and CUDA execution paths. section 9.3

How Quantization and Fast Inference covers it

How deep it goes
Details four ONNX preparation constraints and compares resulting engine performance across two models.
Examples
Compares explicit and implicit INT8 latency for ResNet-18 and BERT-base on an L4 GPU.
What it assumes
Builds on static ONNX quantization with quantize_static; no from-zero treatment is described.
What you can do after
Configure TensorRT-ready QDQ ONNX by choosing supported formats, symmetric activations, operator scope, and layer exclusions.
How it connects
Connects TensorRT engine building with ONNX Q/DQ nodes, static quantization, and INT8 execution.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.