liveBook · Manning

Implicit quantization

As explained in Quantization and Fast Inference

Implicit quantization is a quantization mode in which an inference-engine builder uses calibration information to choose which model layers execute at reduced precision.

Implicit quantization lets a TensorRT builder use calibration data to choose which layers run at INT8. Supply a plain FP32 ONNX model and a calibrator; the builder selects layer precision. Unlike explicit quantization, it does not take fixed placements from Q/DQ nodes. The implicit path is deprecated in TensorRT 10.x. section 9.3

Table 9.2 Engine sizes and execution-precision distribution, L4. Compute layers exclude Reformat/Constant/Shuffle.
Model
Engine
Compute layers
INT8 fraction
Size (MB)
Latency (ms)
ResNet-18
FP32
25
0.0%
47.0
6.66
ResNet-18
FP16
23
0.0%
23.8
2.67
ResNet-18
INT8 implicit
23
95.7%
12.1
1.04
ResNet-18
INT8 explicit
24
83.3%
13.9
1.73
BERT-base
FP32
124
0.0%
438.6
8.77
BERT-base
FP16
86
0.0%
221.1
3.79
BERT-base
INT8 implicit
86
0.0%
221.1
3.77
BERT-base
INT8 explicit
183
97.8%
184.8
2.73

Compare INT8 layer coverage and latency: implicit quantization performs strongly on ResNet-18 but leaves BERT-base entirely outside INT8 compute. section 9.3

How Quantization and Fast Inference covers it

How deep it goes
Explains the FP32 ONNX and calibrator inputs, builder layer selection, and contrast with pinned Q/DQ placements.
Examples
Compares ResNet-18, where implicit INT8 reaches 1.04 ms, with BERT-base, where it achieves no INT8 compute.
What you can do after
Build a TensorRT INT8 engine by providing an FP32 ONNX model and calibration data to the builder.
How it connects
Connects TensorRT engine building with INT8 calibration, ONNX, and explicit quantization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.