TensorRT
TensorRT is NVIDIA’s software development kit for optimizing and executing machine-learning inference on NVIDIA hardware.
LLMs in Production
TensorRT is NVIDIA software for optimizing and executing machine-learning inference on NVIDIA GPUs.
- How it is used
- Optimizes and compiles machine-learning models for inference on NVIDIA GPUs.
- How deep it goes
- Explains graph optimization, quantization, memory techniques, tracing, input shapes, and hardware-specific compilation without detailing the full compiler.
- Examples
- Uses a GPT-2 PyTorch model, traced tokens, configurable shapes, and an NVIDIA Docker container.
- What you can do after
- Apply it via torch_tensorrt to compile a traced PyTorch model for specified NVIDIA GPU hardware.
Quantization and Fast Inference
TensorRT is an NVIDIA software development kit for optimizing and executing neural-network inference on GPUs.
- How it is used
- Selects an NVIDIA-focused runtime for deploying neural-network inference at peak GPU throughput.
- How deep it goes
- Names INT8 and FP8 engine building but does not detail engine configuration or quantization procedures.
- Examples
- Uses an INT8 endpoint with FP32-baseline latency and missing quantized-kernel dispatch to illustrate silent fallback.
- What you can do after
- Apply TensorRT to build INT8 or FP8 inference engines for NVIDIA GPUs.
Read how Quantization and Fast Inference explains TensorRT →
What to know first
- CPUA central processing unit (CPU) is a computer processor that executes program instructions and performs general-purpose computational operations.
- quantizationModel quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
Related concepts
- ONNXuses · ONNX (Open Neural Network Exchange) is an open standard and format for representing and exchanging machine-learning models across frameworks, tools, and runtimes.
- ONNX Runtimealternative to · ONNX Runtime is an open-source, cross-platform runtime for executing and optimizing machine-learning models in the ONNX format.
- tactic selectionuses · Tactic selection is the process of choosing optimized kernel implementations for model operations from an available implementation library.
- GGUFcontrasts with · GGUF is a binary file format for storing machine-learning model tensors and associated metadata, designed to support efficient model loading and inference.
Questions about TensorRT
What is TensorRT?
TensorRT is NVIDIA’s software development kit for optimizing and executing machine-learning inference on NVIDIA hardware.
Which Manning books explain TensorRT?
LLMs in Production and Quantization and Fast Inference.
What should I know before learning TensorRT?
Manning books that teach TensorRT usually introduce CPU and quantization first.
What is the difference between TensorRT and ONNX Runtime?
TensorRT is NVIDIA’s software development kit for optimizing and executing machine-learning inference on NVIDIA hardware. ONNX Runtime is an open-source, cross-platform runtime for executing and optimizing machine-learning models in the ONNX format.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

