liveBook · Manning

Runtime family

As explained in Quantization and Fast Inference

A runtime family is a class of model-execution software sharing a deployment backend or hardware-oriented execution model.

Choose the runtime family after identifying the target hardware. The execution provider then determines which kernels run, and the winning runtime determines the production artifact. section 9.1

Three runtime families cover server-class deployment:

  • ONNX Runtime (ORT) runs ONNX across providers including CPU, CUDA, TensorRT, OpenVINO, DirectML, and ROCm; it suits mixed fleets using one portable artifact. section 9.1
  • TensorRT is NVIDIA’s native compiler; standalone engines favor peak latency, while routing through ORT preserves a unified serving surface. section 9.1
  • OpenVINO is the Intel counterpart, with INT8 support across Xeon CPUs, integrated GPUs, and Core Ultra NPUs. section 9.1

A silent precision fallback

A quantized artifact may execute in FP32 without an error if the runtime lacks a kernel for its target dtype on the deployment hardware. Verify the executing dtype on the deploy host. section 9.1 section 9.6

Figure 9.1 Hardware selects the runtime family, and the execution provider selects which kernels dispatch. The leaf node is the production artifact. The two routes to TensorRT and OpenVINO, standalone versus via ORT, trade peak latency against a unified serving surface across heterogeneous fleets. Branches marked Chapter 10 and Chapter 11 point to regimes covered there.

Trace the decision from hardware to runtime family, execution provider, and production artifact; compare standalone and via-ORT routes. section 9.1

How Quantization and Fast Inference covers it

How deep it goes
Defines three server runtime families and compares deployment routes, while leaving detailed quantization recipes to separate decisions.
Examples
Compares ONNX Runtime, TensorRT, and OpenVINO, including TensorRT .plan and OpenVINO IR production artifacts.
What it assumes
Assumes a target hardware class and distinguishes runtime family from execution provider and production artifact.
What you can do after
Configure a deployment path by matching hardware to a runtime, selecting execution providers, and verifying actual precision on-host.
How it connects
Connects execution providers, ONNX, quantization, production artifacts, TensorRT, OpenVINO, and hardware-specific kernels.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.