liveBook · Manning

TensorRT

As explained in Quantization and Fast Inference

TensorRT is an NVIDIA software development kit for optimizing and executing neural-network inference on GPUs.

TensorRT is an NVIDIA SDK for optimizing and executing neural-network inference on GPUs. It is positioned for peak throughput on NVIDIA hardware and for building INT8 and FP8 engines. In deployment, verify that quantized kernels actually run: a quantized artifact can still serve through a silent FP32 fallback. section 9.0

Quantized artifact, unquantized execution

An INT8 endpoint shows p99 latency at the FP32 baseline, while GPU dispatch counters show no quantized kernels. The artifact alone does not prove the serving path uses INT8. section 9.0

How Quantization and Fast Inference covers it

How deep it goes
Names INT8 and FP8 engine building but does not detail engine configuration or quantization procedures.
Examples
Uses an INT8 endpoint with FP32-baseline latency and missing quantized-kernel dispatch to illustrate silent fallback.
What you can do after
Apply TensorRT to build INT8 or FP8 inference engines for NVIDIA GPUs.
How it connects
Compares TensorRT with ONNX Runtime and OpenVINO as runtime choices for different hardware targets.

Also explained in

Compare how 2 books explain TensorRT →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.