liveBook · Manning

Triton model repository

As explained in Quantization and Fast Inference

A Triton model repository is a directory structure containing model artifacts and configuration files that Triton uses to load and serve models.

A Triton model repository packages each artifact–runtime pair in its own variant directory, giving the serving runtime a configuration alongside the model files. It makes deployment choices and runtime compatibility explicit rather than relying on the artifact alone. section 9.5

Repository layout

A variant follows `packages/<model>/triton_model_repo/<variant>/{1/, config.pbtxt}`. Example variants include `bert_sst2_ort_int8` and `bert_sst2_openvino_int8`. Configuration can specify input and output shapes, instance groups, and dynamic batching; precision is generally represented in the artifact graph, not the config. section 9.5

Figure 9.10 Packaging verification matrix on an NVIDIA L4 host. Rows are artifact kinds; columns are runtimes. Solid cells executed at the claimed precision (the OpenVINO Core CPU column reports per-op precision fractions; ORT cells verify load and run, since ORT's Python API does not expose per-op precision). Hatched cells loaded but fell back. ResNet INT8 × ORT TRT EP routed 19 nodes to CUDA EP because TensorRT's parser rejects ORT's INT32 zero-point format on Conv DQ; BERT INT8 × ORT CUDA EP is the 9.2 anti-pattern resurfacing. Filled cells are runtime, hardware, or artifact absent; information about the deploy target, not bugs.

Compare artifact–runtime verification results across hardware hosts; the matrix distinguishes successful execution, fallback, and unavailable combinations. section 9.5

How Quantization and Fast Inference covers it

How deep it goes
Shows the variant directory layout and configuration role, then validates artifact–runtime pairs on target hosts.
Examples
Uses BERT INT8 variants such as `bert_sst2_ort_int8` and `bert_sst2_openvino_int8`.
What you can do after
Build a Triton repository of artifact–runtime variants and verify their loading and execution on available hosts.
How it connects
Connects model artifacts, Triton configuration, ONNX Runtime, TensorRT, OpenVINO, and dynamic batching.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.