Explicit quantization pins quantization placements in ONNX with Q/DQ nodes, which TensorRT must honor when building an INT8 engine. It is the recommended TensorRT path, but it does not guarantee the lowest latency for every model. section 9.3
Prepare the ONNX artifact for TensorRT by:
- Using QDQ format rather than QOperator. section 9.3
- Setting activations to symmetric quantization. section 9.3
- Restricting quantized operators to Conv or MatMul, and excluding quantization-sensitive layers near inputs and outputs. section 9.3
Latency depends on the model
On an L4, explicit INT8 runs ResNet-18 at 1.73 ms, slower than implicit INT8 at 1.04 ms; BERT-base runs at 2.73 ms, while implicit INT8 is near FP16 latency. section 9.3
Figure 9.7 BERT-base SST-2 latency on L4, batch 8, seq 128. Explicit INT8 leads at 2.73 ms; implicit INT8 lands within measurement noise of FP16. Dotted lines mark Table 9.1's CUDA EP INT8 (14.4 ms) and FP32-optimized (9 ms) references. TRT's INT8 explicit closes the gap section 9.2 identified.

Compare the BERT-base latency results across TensorRT and CUDA execution paths. section 9.3
