9 Shipping with the Right Toolchain
This chapter covers
- Matching runtimes to target hardware
- Deploying through ONNX Runtime execution providers
- Building TensorRT INT8 and FP8 engines
- Quantizing for Intel CPUs with OpenVINO
- Catching silent FP32 fallbacks
- Packaging artifacts with precision metadata
We built quantized models in memory. The result was an artifact that runs at 8 bits or below on a developer's machine. Now we need to discuss the gap between runs on a developer's machine and runs in production, where the silent failures happen.
This lesson is for the inference engineer on the hook for the path from a quantized checkpoint to a serving endpoint, the person who takes the model from the research team and owns latency, dtype verification, packaging, and rollback for it.
The path passes through graph formats, runtimes, and execution providers, and a wrong call at any stage leaves the kernels running in FP32 at serving time with no error, no warning, no alert. Picture the on-call version. The rollout dashboard shows the INT8 endpoint live, but p99 latency never leaves the FP32 baseline and the GPU dispatch counters show the quantized kernels were never called. Nothing in the artifact on disk reveals which side is right; the file is quantized, the execution is not.