liveBook · Manning

TensorRT engine

As explained in Quantization and Fast Inference

A TensorRT engine is a compiled model artifact optimized for execution by TensorRT on supported hardware.

A TensorRT engine is a production artifact compiled for serving silicon. It differs from portable ONNX, which can be loaded by multiple runtimes. section 9.1

Choose the deployment route to match the fleet:

  • For a single-SKU deployment at scale, serve a standalone engine through the vendor API to favor peak latency. section 9.1
  • For a heterogeneous fleet, keep ONNX Runtime serving and use its TensorRT execution provider for compatible subgraphs. section 9.1
Figure 9.1 Hardware selects the runtime family, and the execution provider selects which kernels dispatch. The leaf node is the production artifact. The two routes to TensorRT and OpenVINO, standalone versus via ORT, trade peak latency against a unified serving surface across heterogeneous fleets. Branches marked Chapter 10 and Chapter 11 point to regimes covered there.

Trace how hardware and runtime choice lead to a production artifact, and compare standalone serving with the ONNX Runtime route. section 9.1

How Quantization and Fast Inference covers it

How deep it goes
Compares standalone and ONNX Runtime deployment routes and their latency-versus-serving-surface tradeoffs.
Examples
Contrasts .plan engines with portable ONNX and illustrates standalone versus ONNX Runtime deployment.
What you can do after
Operate/configure deployment by choosing standalone serving for single-SKU scale or ONNX Runtime for heterogeneous fleets.
How it connects
Connects TensorRT engines with ONNX, ONNX Runtime, and execution providers.

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.