Chapter 9: Shipping with the Right Toolchain
This introduction explains why a quantized model that works locally may still run at FP32 in production, making serving-time verification essential.
After this chapter you will be able to
- Quantized artifact versus execution. Explain how a quantized checkpoint can reach a serving endpoint yet execute with FP32 kernels, and why the artifact alone cannot verify the precision used at runtime.
- Hardware Driven Runtime Selection. Explain how target hardware determines the runtime family and how an execution provider determines which kernels dispatch. Distinguish ORT's mixed-fleet role from TensorRT's NVIDIA focus and OpenVINO's Intel focus.
- Portable Versus Production Artifacts. Distinguish the vendor-neutral ONNX graph handed between teams from the artifact consumed by the chosen production runtime, while noting that ORT can serve quantized ONNX directly.
- Runtime Precision Verification. Explain why inspecting a quantized file cannot establish its serving precision and describe what must be checked in the target runtime instead.
- Task-Aware Export and Quantization. Distinguish Optimum's task-aware export and quantization classes from the raw ONNX Runtime quantization API and the separate transformer optimizer.
- Transformer Quantization Defaults. Explain why dynamic INT8 outperformed the default static INT8 BERT path here, and describe the static overrides recommended for this graph.
- Transformer-Aware Graph Fusion. Explain what the transformer-aware optimizer fuses and why its pass is separate from both operator-local quantization and session-init optimization.
- Format and Provider Matching. Distinguish QOperator from QDQ and justify the section's CPU and CUDA execution-provider choices using the observed latency and kernel behavior.
- Two TensorRT INT8 paths. Distinguish calibrator-based implicit quantization from Q/DQ-based explicit quantization, and explain why the choice changes the engine TensorRT builds.
- Explicit ONNX compatibility settings. Describe the four changes to the ONNX Runtime quantization call needed to make explicit Q/DQ ONNX consumable by TensorRT, including the incompatibilities they avoid.
- ResNet fusion and latency. Explain why implicit INT8 is faster than explicit INT8 for the ResNet-18 example despite both engines advertising INT8.
- BERT calibration coverage failure. Trace why the implicit BERT-base engine performs no INT8 compute and how explicit Q/DQ avoids that outcome.
- Actual execution precision checks. Explain how the section checks whether a TensorRT engine actually executes in INT8 rather than relying on its advertised precision.
- Two OpenVINO deployment routes. Describe how the standalone NNCF route and the ONNX Runtime OpenVINO Execution Provider route produce different artifacts while using the same OpenVINO CPU plugin. Explain why an existing serving stack can influence the choice.
- Artifact recipes and model latency. Explain why neither INT8 route is consistently faster: their quantization recipes differ, and ResNet-18 favors native while BERT-base favors via-ORT on the measured Sapphire Rapids CPU.
- Mixed precision across CPU generations. Explain why an INT8 OpenVINO graph can execute partly in BF16 on Sapphire Rapids but partly in FP32 on an older CPU.
- Executed precision verification. Describe how to check hardware precision support and inspect the precision OpenVINO actually executes. Explain why an overall INT8-op percentage cannot be compared directly with TensorRT's.
- Artifact manifest contract. Explain why an executable model file needs a manifest sidecar and what the sidecar records.
- Triton runtime packaging. Describe how the Triton model repository represents artifact-runtime pairs and where its precision and execution-provider choices are specified.
- Deployability verification matrix. Explain what the host verification matrix tests, what its four cell states mean, and how a deploy gate uses its report.
- NVIDIA QDQ execution pitfalls. Explain why the section marks two NVIDIA QDQ pairings as demoted despite successful loading, and why one successful model does not validate another.
- Host-dependent precision mix. Distinguish the quantizer-determined INT8 share from the host-dependent BF16 execution observed when comparing the L4 and Sapphire Rapids matrices.
What it builds on
- Export to ONNX and drive ONNX Runtime quantization (chapter 6) · Earlier ONNX Runtime quantization provides context for serving an ONNX graph.
- Estimate ranges (absolute max, percentile, and MSE) (chapter 4) · Calibration methods and the ranges they estimate.
Section by section
Introduction
This introduction explains why a quantized model that works locally may still run at FP32 in production, making serving-time verification essential.
- Quantized artifact versus execution: Explain how a quantized checkpoint can reach a serving endpoint yet execute with FP32 kernels, and why the artifact alone cannot verify the precision used at runtime.
Why it matters: A successful quantized export does not guarantee the latency or kernel execution expected in production.
Key terms: quantized checkpoint, graph formats, runtimes, execution providers, FP32, dtype verification
9.1 Choose graphs, runtimes, and artifacts for deployment
This section shows how to choose a deployment runtime and artifact for the target hardware, then verify that serving actually uses the intended numeric precision.
- Hardware Driven Runtime Selection: Explain how target hardware determines the runtime family and how an execution provider determines which kernels dispatch. Distinguish ORT's mixed-fleet role from TensorRT's NVIDIA focus and OpenVINO's Intel focus.
Why it matters: The right quantized model file cannot deliver the intended execution if the serving hardware and runtime dispatch the wrong kernels. - Standalone And ORT Routes: Distinguish serving through standalone TensorRT or OpenVINO from accessing those vendor runtimes through ORT execution providers, including the deployment situation each route favors.
Why it matters: The route trades a vendor-specific serving path for the convenience of retaining one ORT-based service. - Portable Versus Production Artifacts: Distinguish the vendor-neutral ONNX graph handed between teams from the artifact consumed by the chosen production runtime, while noting that ORT can serve quantized ONNX directly.
Why it matters: Treating an exchange file as proof of a deployable, quantized execution path can conceal an FP32 fallback. - Runtime Precision Verification: Explain why inspecting a quantized file cannot establish its serving precision and describe what must be checked in the target runtime instead.
Why it matters: Checking actual kernel precision catches performance-relevant fallbacks that a correct-looking quantized file will not reveal.
Key terms: target hardware, runtime family, execution provider, ONNX Runtime, TensorRT, OpenVINO, standalone, via ORT, trtexec, ovc
9.2 Use Optimum and ONNX Runtime quantizers the right way
This section assembles and benchmarks an Optimum and ONNX Runtime deployment recipe, showing how quantization defaults, graph fusion, execution providers, calibration, and session settings affect BERT inference.
- Task-Aware Export and Quantization: Distinguish Optimum's task-aware export and quantization classes from the raw ONNX Runtime quantization API and the separate transformer optimizer.
Why it matters: Choosing the right entry point avoids treating export, quantization, and graph optimization as interchangeable steps. - Transformer Quantization Defaults: Explain why dynamic INT8 outperformed the default static INT8 BERT path here, and describe the static overrides recommended for this graph.
Why it matters: A static INT8 build can be slower, less accurate, or fail outright when transformer defaults are left unchecked. - Transformer-Aware Graph Fusion: Explain what the transformer-aware optimizer fuses and why its pass is separate from both operator-local quantization and session-init optimization.
Why it matters: Graph fusion can improve latency before any INT8 gain is considered. - Format and Provider Matching: Distinguish QOperator from QDQ and justify the section's CPU and CUDA execution-provider choices using the observed latency and kernel behavior.
Why it matters: An INT8 artifact is not automatically a faster deployment artifact on every execution provider. - Calibration Choice by Activation: Explain what changes between static-INT8 calibration methods and why the ResNet-18 comparison did not establish a universal best method.
Why it matters: Calibration should be chosen for the activation distribution, not from a small apparent latency difference between otherwise equivalent graphs. - Session Knobs and Bottlenecks: Explain why io_binding and tuned SessionOptions showed no measurable gain on the already-optimized BERT batch-classification graph.
Why it matters: Measure optimization knobs against the bottleneck that remains, rather than assuming every useful knob compounds on an optimized graph.
Key terms: optimum.onnxruntime, ORTModelForSequenceClassification, ORTQuantizer, onnxruntime.quantization, onnxruntime.transformers.optimizer, dynamic quantization, static INT8, MatMulConstBOnly, Gather, operators_to_quantize
9.3 Build TensorRT engines with real integer execution
This section compares TensorRT’s implicit and explicit INT8 builds, showing why an engine labeled INT8 may run mostly—or entirely—at higher precision.
- Two TensorRT INT8 paths: Distinguish calibrator-based implicit quantization from Q/DQ-based explicit quantization, and explain why the choice changes the engine TensorRT builds.
Why it matters: The same INT8 build flag does not guarantee the same placements or performance. - Explicit ONNX compatibility settings: Describe the four changes to the ONNX Runtime quantization call needed to make explicit Q/DQ ONNX consumable by TensorRT, including the incompatibilities they avoid.
Why it matters: An explicit INT8 engine cannot be built from just any ONNX Runtime quantized artifact. - ResNet fusion and latency: Explain why implicit INT8 is faster than explicit INT8 for the ResNet-18 example despite both engines advertising INT8.
Why it matters: Precise quantization placement can cost latency when it prevents a useful fusion. - BERT calibration coverage failure: Trace why the implicit BERT-base engine performs no INT8 compute and how explicit Q/DQ avoids that outcome.
Why it matters: An INT8-labeled transformer engine can silently behave like a higher-precision engine if calibration misses scales. - Actual execution precision checks: Explain how the section checks whether a TensorRT engine actually executes in INT8 rather than relying on its advertised precision.
Why it matters: Checking the built engine prevents an INT8 label from being mistaken for an INT8 speedup.
Key terms: implicit quantization, explicit quantization, calibrator, Q/DQ nodes, QuantFormat.QDQ, ActivationSymmetric, op_types_to_quantize, nodes_to_exclude, stem, INT8 supernode
9.4 Deploy on Intel CPU with OpenVINO and NNCF
This section compares two OpenVINO deployment routes on Intel CPUs and shows how artifacts, hardware capabilities, and executed precision affect latency.
- Two OpenVINO deployment routes: Describe how the standalone NNCF route and the ONNX Runtime OpenVINO Execution Provider route produce different artifacts while using the same OpenVINO CPU plugin. Explain why an existing serving stack can influence the choice.
Why it matters: The route determines what you ship and whether you can keep an existing ORT serving path. - Artifact recipes and model latency: Explain why neither INT8 route is consistently faster: their quantization recipes differ, and ResNet-18 favors native while BERT-base favors via-ORT on the measured Sapphire Rapids CPU.
Why it matters: Benchmarking both artifacts avoids assuming that a shared runtime guarantees equal performance. - Mixed precision across CPU generations: Explain why an INT8 OpenVINO graph can execute partly in BF16 on Sapphire Rapids but partly in FP32 on an older CPU.
Why it matters: A model's INT8 label alone cannot predict its executed precision or latency on another CPU. - Executed precision verification: Describe how to check hardware precision support and inspect the precision OpenVINO actually executes. Explain why an overall INT8-op percentage cannot be compared directly with TensorRT's.
Why it matters: These checks catch silent precision fallback and prevent misleading comparisons between runtime reports. - BF16 deployment trade-off: Justify choosing BF16 instead of calibrated INT8 when BF16 already meets the latency budget on supported hardware.
Why it matters: The fastest measured format is not necessarily the cheapest one to deploy and maintain.
Key terms: Standalone path, NNCF, IR, OpenVINOExecutionProvider, OpenVINO CPU plugin, MIXED preset, per-channel, activation symmetry, op scope, QDQ ONNX
9.5 Package models for serving and distribution
This section turns quantized model files into deployable packages by pairing them with manifests and runtime configurations, then verifying what actually executes on each host.
- Artifact manifest contract: Explain why an executable model file needs a manifest sidecar and what the sidecar records.
Why it matters: A valid model file alone does not tell a deployment team whether it is the right file for a host or how it was produced. - Triton runtime packaging: Describe how the Triton model repository represents artifact-runtime pairs and where its precision and execution-provider choices are specified.
Why it matters: Packaging must select the intended runtime without pretending a config file determines the model's quantized precision. - Deployability verification matrix: Explain what the host verification matrix tests, what its four cell states mean, and how a deploy gate uses its report.
Why it matters: The gate can distinguish a working INT8 deployment from a model that merely loads and silently runs at higher precision. - NVIDIA QDQ execution pitfalls: Explain why the section marks two NVIDIA QDQ pairings as demoted despite successful loading, and why one successful model does not validate another.
Why it matters: A successful load—or even success for a different architecture—can conceal floating-point execution behind an INT8 label. - Host-dependent precision mix: Distinguish the quantizer-determined INT8 share from the host-dependent BF16 execution observed when comparing the L4 and Sapphire Rapids matrices.
Why it matters: Comparing hosts requires separating what the quantizer selected from what the hardware and runtime dispatch. - Matrix evidence boundaries: State what the verification matrix cannot establish and how a missing required TensorRT plan affects the deploy gate.
Why it matters: A precision check is valuable evidence, but it cannot substitute for kernel diagnostics, latency measurements, or a required artifact.
Key terms: manifest sidecar, calibration fingerprint, runtime compatible, SHA-256, Triton model-repo, config.pbtxt, runtimePrecision, ORT-OV-EP, execution_accelerators, verification matrix
9.6 Summary
Where this is used later
- (chapter intro) (chapter 10) · Introduces the distinction between deployment graphs, runtimes, and artifacts.
- The packing is not uniform (chapter 10) · The earlier sensitivity-driven pattern of excluding selected nodes from quantization.
- Phone, SBC, and small-box tiers (chapter 11) · The TensorRT toolchain used for Jetson.
- Ship a four-bit large language model on a single GPU with tests (chapter 12) · Shipping a tested GPU model depends on checking the artifact and its deployment assumptions.
- Metadata makes the file self-describing (chapter 10) · The separate manifest-sidecar packaging pattern used with ORT and TensorRT.
Concepts explained in this chapter
- OpenVINO IROpenVINO IR is an intermediate model representation stored in XML and binary files for use with OpenVINO tools and runtimes.
- verification matrixA verification matrix is a structured assessment of model-artifact and runtime combinations, recording whether each combination loads and executes as expected.
- QOperatorQOperator is an ONNX quantization format that replaces targeted operators with explicit integer operators.
- min-max calibrationMin-max calibration uses the minimum and maximum observed values to determine a quantization range and its parameters.
- Triton model repositoryA Triton model repository is a directory structure containing model artifacts and configuration files that Triton uses to load and serve models.
- runtime contractA runtime contract specifies the compatibility and execution expectations between a model artifact and the runtime that loads it.
- explicit quantizationExplicit quantization is a quantization mode in which quantization locations and parameters are represented in the model graph and honored by the inference-engine builder.
- implicit quantizationImplicit quantization is a quantization mode in which an inference-engine builder uses calibration information to choose which model layers execute at reduced precision.
- operator coverageThe proportion or extent of a model's operators affected by an optimization or quantization method.
- NNCFNNCF is OpenVINO's neural network compression framework for optimizing models, including through quantization.
- OpenVINO CPU pluginThe OpenVINO CPU plugin is the OpenVINO backend that compiles and executes model graphs on CPUs.
- OpenVINO runtime model inspectorThe OpenVINO runtime model inspector examines a compiled runtime graph and reports the precision used by its operations.
- OpenVINO through ONNX RuntimeOpenVINO through ONNX Runtime is a deployment pattern that retains ONNX Runtime serving code and uses the OpenVINO execution provider.
- OpenVINOExecutionProviderThe OpenVINO execution provider is an ONNX Runtime backend that delegates model operations to OpenVINO.
- standalone OpenVINO deploymentStandalone OpenVINO deployment converts a model to IR, quantizes it with NNCF, and serves it through OpenVINO.
- execution providerAn execution provider is an ONNX Runtime backend that assigns and executes model-graph operations on a particular hardware or software target.
- runtime familyA runtime family is a class of model-execution software sharing a deployment backend or hardware-oriented execution model.
- model artifactA model artifact is a packaged, deployable representation of a machine-learning model and the information needed to execute it.
- static quantizationStatic quantization determines quantization ranges during calibration before deployment and keeps the resulting parameters fixed at runtime.
- ONNX RuntimeONNX Runtime is an inference engine for executing models represented in the Open Neural Network Exchange format.
- OpenVINOOpenVINO is an Intel toolkit for optimizing and deploying machine-learning models across diverse hardware targets.
- TensorRTTensorRT is an NVIDIA software development kit for optimizing and executing neural-network inference on GPUs.
- tactic selectionTactic selection is the process of choosing among alternative implementations for neural-network operations on a target device.
- TensorRT engineA TensorRT engine is a compiled model artifact optimized for execution by TensorRT on supported hardware.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
