liveBook · Manning

TensorRT

TensorRT is NVIDIA’s software development kit for optimizing and executing machine-learning inference on NVIDIA hardware.

How 2 Manning books explain it

LLMs in Production

TensorRT is NVIDIA software for optimizing and executing machine-learning inference on NVIDIA GPUs.

How it is used
Optimizes and compiles machine-learning models for inference on NVIDIA GPUs.
How deep it goes
Explains graph optimization, quantization, memory techniques, tracing, input shapes, and hardware-specific compilation without detailing the full compiler.
Examples
Uses a GPT-2 PyTorch model, traced tokens, configurable shapes, and an NVIDIA Docker container.
What you can do after
Apply it via torch_tensorrt to compile a traced PyTorch model for specified NVIDIA GPU hardware.

Read how LLMs in Production explains TensorRT →

Quantization and Fast Inference

TensorRT is an NVIDIA software development kit for optimizing and executing neural-network inference on GPUs.

How it is used
Selects an NVIDIA-focused runtime for deploying neural-network inference at peak GPU throughput.
How deep it goes
Names INT8 and FP8 engine building but does not detail engine configuration or quantization procedures.
Examples
Uses an INT8 endpoint with FP32-baseline latency and missing quantized-kernel dispatch to illustrate silent fallback.
What you can do after
Apply TensorRT to build INT8 or FP8 inference engines for NVIDIA GPUs.

Read how Quantization and Fast Inference explains TensorRT →

What to know first

Related concepts

Questions about TensorRT

What is TensorRT?

TensorRT is NVIDIA’s software development kit for optimizing and executing machine-learning inference on NVIDIA hardware.

Which Manning books explain TensorRT?

LLMs in Production and Quantization and Fast Inference.

What should I know before learning TensorRT?

Manning books that teach TensorRT usually introduce CPU and quantization first.

What is the difference between TensorRT and ONNX Runtime?

TensorRT is NVIDIA’s software development kit for optimizing and executing machine-learning inference on NVIDIA hardware. ONNX Runtime is an open-source, cross-platform runtime for executing and optimizing machine-learning models in the ONNX format.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.