How deeply does LLMs in Production cover TensorRT?

Explains graph optimization, quantization, memory techniques, tracing, input shapes, and hardware-specific compilation without detailing the full compiler.

TensorRT

As explained in LLMs in Production

TensorRT is NVIDIA software for optimizing and executing machine-learning inference on NVIDIA GPUs.

TensorRT turns a trained model into an NVIDIA GPU-specific inference engine. It combines graph optimization, INT8 quantization, and memory techniques to reduce runtime cost, but the resulting engine must match the hardware where it will run. section 6.1.1

A practical TensorRT workflow typically involves:

Loading a PyTorch model and representative input, then tracing it into TorchScript so its operations can be optimized. section 6.1.1
Defining static or dynamic input shapes and datatypes in torch_tensorrt before compilation. section 6.1.1

How LLMs in Production covers it

liveBook manning.com

Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

ONNX Runtimeused by · ONNX Runtime is an open-source, cross-platform inference engine for executing and optimizing machine-learning models represented in the ONNX format.
ONNXuses · ONNX is an open standard for representing machine-learning models in a portable format that supports exchange and execution across frameworks and runtimes.
edge deviceuses · An edge device is a computing device located near data sources, users, or operations that processes or uses data close to where it is generated or consumed.
quantizationuses · Quantization maps numerical values to a finite set of discrete representations, often using reduced precision to lower model storage or computation requirements.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.