liveBook · Manning

Verification matrix

As explained in Quantization and Fast Inference

A verification matrix is a structured assessment of model-artifact and runtime combinations, recording whether each combination loads and executes as expected.

A verification matrix tests whether each model artifact and runtime combination loads and executes as expected on a target host. Run it before deployment: a file that exists is not necessarily loadable, supported by the target hardware, or executed at its claimed precision. section 9.5

Read each cell as a runtime contract check:

  • A solid cell means the pair executed at the claimed precision; a hatched cell means it loaded but fell back. section 9.5
  • A filled cell means the runtime, hardware, or artifact was absent. Treat that as information about the deploy target, not automatically as a bug. section 9.5
  • Repeat the checks on the intended host: moving artifacts to different silicon can change which combinations are reachable. section 9.6

A precision mismatch can hide behind a successful run

On an NVIDIA L4, BERT INT8 QDQ ONNX with the ORT CUDA Execution Provider loaded and ran, but was demoted for partial INT8 execution. ResNet INT8 with the ORT TensorRT provider also fell back: 19 nodes routed to CUDA because TensorRT rejected the INT32 zero-point format on Conv DQ. section 9.5

Figure 9.10 Packaging verification matrix on an NVIDIA L4 host. Rows are artifact kinds; columns are runtimes. Solid cells executed at the claimed precision (the OpenVINO Core CPU column reports per-op precision fractions; ORT cells verify load and run, since ORT's Python API does not expose per-op precision). Hatched cells loaded but fell back. ResNet INT8 × ORT TRT EP routed 19 nodes to CUDA EP because TensorRT's parser rejects ORT's INT32 zero-point format on Conv DQ; BERT INT8 × ORT CUDA EP is the 9.2 anti-pattern resurfacing. Filled cells are runtime, hardware, or artifact absent; information about the deploy target, not bugs.

Compare artifact rows and runtime columns on the L4 host, including successful execution, fallback, and unavailable combinations. section 9.5

How Quantization and Fast Inference covers it

How deep it goes
Separates load-and-run checks from precision evidence: OpenVINO reports per-op fractions, while ORT cells verify load and run only.
Examples
Compares INT8 BERT and ResNet artifact-runtime pairs on NVIDIA L4 and Sapphire Rapids hosts, including fallback and blocked outcomes.
What you can do after
Apply it via ch9_packaging_serving.py to test artifact-runtime pairs on the host before deployment.
How it connects
Connects model artifacts, inference runtimes, target hardware, quantization, and deployment gates.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.