liveBook · Manning

Tactic selection

As explained in Quantization and Fast Inference

Tactic selection is the process of choosing among alternative implementations for neural-network operations on a target device.

Tactic selection chooses operation implementations from NVIDIA’s kernel library while TensorRT builds an engine for a target GPU. The build selects tactics ahead of time, can fuse Q/DQ pairs into INT8 supernodes, and produces a serialized plan. section 9.3

The resulting engine depends on the quantization path as well as the target and model. TensorRT supports implicit quantization, where it chooses INT8 layers, and explicit Q/DQ quantization, where it must honor placements encoded in ONNX. section 9.3

Quantization choices produce model-dependent results

On an L4, implicit INT8 runs ResNet-18 at 1.04 ms versus 1.73 ms for explicit INT8. For BERT-base, explicit INT8 leads at 2.73 ms, while implicit INT8 performs near FP16. section 9.3

How Quantization and Fast Inference covers it

How deep it goes
Describes ahead-of-time tactic selection and engine compilation, but does not detail the selection algorithm.
Examples
Compares implicit and explicit INT8 engine latency for ResNet-18 and BERT-base on an L4.
What it assumes
Requires an ONNX model and a target NVIDIA GPU for TensorRT engine compilation.
What you can do after
Operate TensorRT to build target-specific engines, choosing between implicit quantization and explicit Q/DQ placements.
How it connects
Connects TensorRT tactic selection with ONNX, ONNX Runtime, CUDA GPUs, and INT8 quantization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.