liveBook · Manning

Chapter 9: Shipping with the Right Toolchain

What you will learn in chapter 9 of Quantization and Fast Inference

This introduction explains why a quantized model that works locally may still run at FP32 in production, making serving-time verification essential.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction explains why a quantized model that works locally may still run at FP32 in production, making serving-time verification essential.

Key terms: quantized checkpoint, graph formats, runtimes, execution providers, FP32, dtype verification

9.1 Choose graphs, runtimes, and artifacts for deployment

This section shows how to choose a deployment runtime and artifact for the target hardware, then verify that serving actually uses the intended numeric precision.

Key terms: target hardware, runtime family, execution provider, ONNX Runtime, TensorRT, OpenVINO, standalone, via ORT, trtexec, ovc

9.2 Use Optimum and ONNX Runtime quantizers the right way

This section assembles and benchmarks an Optimum and ONNX Runtime deployment recipe, showing how quantization defaults, graph fusion, execution providers, calibration, and session settings affect BERT inference.

Key terms: optimum.onnxruntime, ORTModelForSequenceClassification, ORTQuantizer, onnxruntime.quantization, onnxruntime.transformers.optimizer, dynamic quantization, static INT8, MatMulConstBOnly, Gather, operators_to_quantize

9.3 Build TensorRT engines with real integer execution

This section compares TensorRT’s implicit and explicit INT8 builds, showing why an engine labeled INT8 may run mostly—or entirely—at higher precision.

Key terms: implicit quantization, explicit quantization, calibrator, Q/DQ nodes, QuantFormat.QDQ, ActivationSymmetric, op_types_to_quantize, nodes_to_exclude, stem, INT8 supernode

9.4 Deploy on Intel CPU with OpenVINO and NNCF

This section compares two OpenVINO deployment routes on Intel CPUs and shows how artifacts, hardware capabilities, and executed precision affect latency.

Key terms: Standalone path, NNCF, IR, OpenVINOExecutionProvider, OpenVINO CPU plugin, MIXED preset, per-channel, activation symmetry, op scope, QDQ ONNX

9.5 Package models for serving and distribution

This section turns quantized model files into deployable packages by pairing them with manifests and runtime configurations, then verifying what actually executes on each host.

Key terms: manifest sidecar, calibration fingerprint, runtime compatible, SHA-256, Triton model-repo, config.pbtxt, runtimePrecision, ORT-OV-EP, execution_accelerators, verification matrix

9.6 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 8: Pushing into Low Bits Safely Chapter 10: Running on CPUs and Using GGUF →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.