liveBook · Manning

Runtime contract

As explained in Quantization and Fast Inference

A runtime contract specifies the compatibility and execution expectations between a model artifact and the runtime that loads it.

A runtime contract states which model-artifact and runtime pairs are expected to load and execute, and under what conditions. The artifact graph may encode precision, scales, and zero points, but not the calibration fingerprint, quantization recipe, or minimum compute capability; a manifest sidecar records these alongside hardware and runtime requirements. section 9.5

Make the contract operational by:

  • Keeping a Triton model-repository directory for each artifact-runtime pair. Configuration usually describes inputs, outputs, batching, and instances; ORT's OpenVINO execution-provider selection is configured there. section 9.5
  • Smoke-testing each applicable pair on the target host, recording whether it loads, which providers resolve, and whether execution uses the claimed precision. section 9.5

Loading is not sufficient

On an NVIDIA L4, BERT INT8 QDQ ONNX loaded through ORT's CUDA execution provider but was demoted: execution was partly INT8 and partly FP32. It measured 14.4 ms, versus 8.96 ms for optimized FP32, because CUDA EP coverage was incomplete. section 9.5

Figure 9.10 Packaging verification matrix on an NVIDIA L4 host. Rows are artifact kinds; columns are runtimes. Solid cells executed at the claimed precision (the OpenVINO Core CPU column reports per-op precision fractions; ORT cells verify load and run, since ORT's Python API does not expose per-op precision). Hatched cells loaded but fell back. ResNet INT8 × ORT TRT EP routed 19 nodes to CUDA EP because TensorRT's parser rejects ORT's INT32 zero-point format on Conv DQ; BERT INT8 × ORT CUDA EP is the 9.2 anti-pattern resurfacing. Filled cells are runtime, hardware, or artifact absent; information about the deploy target, not bugs.

Compare artifact-runtime pairs by whether they execute at the claimed precision, fall back, or are unavailable on the host. section 9.5

How Quantization and Fast Inference covers it

How deep it goes
Combines artifact sidecars, pair-specific Triton configuration, and host smoke tests; distinguishes loading from claimed-precision execution.
Examples
Uses QDQ INT8 artifacts, TensorRT plans, OpenVINO IR, and BERT and ResNet host verification matrices.
What you can do after
Diagnose whether an artifact-runtime pair loads and executes at its claimed precision on a specified host.
How it connects
Connects model packaging with quantization metadata, ONNX Runtime execution providers, Triton, OpenVINO, and TensorRT.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.