Implicit quantization lets a TensorRT builder use calibration data to choose which layers run at INT8. Supply a plain FP32 ONNX model and a calibrator; the builder selects layer precision. Unlike explicit quantization, it does not take fixed placements from Q/DQ nodes. The implicit path is deprecated in TensorRT 10.x. section 9.3
Table 9.2 Engine sizes and execution-precision distribution, L4. Compute layers exclude Reformat/Constant/Shuffle.
|
Model
|
Engine
|
Compute layers
|
INT8 fraction
|
Size (MB)
|
Latency (ms)
|
|
ResNet-18
|
FP32
|
25
|
0.0%
|
47.0
|
6.66
|
|
ResNet-18
|
FP16
|
23
|
0.0%
|
23.8
|
2.67
|
|
ResNet-18
|
INT8 implicit
|
23
|
95.7%
|
12.1
|
1.04
|
|
ResNet-18
|
INT8 explicit
|
24
|
83.3%
|
13.9
|
1.73
|
|
BERT-base
|
FP32
|
124
|
0.0%
|
438.6
|
8.77
|
|
BERT-base
|
FP16
|
86
|
0.0%
|
221.1
|
3.79
|
|
BERT-base
|
INT8 implicit
|
86
|
0.0%
|
221.1
|
3.77
|
|
BERT-base
|
INT8 explicit
|
183
|
97.8%
|
184.8
|
2.73
|
Compare INT8 layer coverage and latency: implicit quantization performs strongly on ResNet-18 but leaves BERT-base entirely outside INT8 compute. section 9.3
