NVIDIA Triton Inference Service

As explained in Designing Deep Learning Systems: A guide for software engineers

NVIDIA Triton Inference Server is an open-source inference-serving system that supports models from multiple machine-learning frameworks and hardware backends.

NVIDIA Triton Inference Server is an open-source inference-serving system for deploying models across CPUs, GPUs, cloud infrastructure, data centers, and edge applications. It accepts remote inference requests over REST or gRPC and can also be embedded at the edge through a C API. section 7.4.3

Triton separates request handling from model execution. A filesystem-based model repository supplies model files; each model receives a configurable scheduler that can batch requests before passing them to a framework-specific backend. section 7.4.3

Use Triton when the serving system needs:

• One serving platform for models from TensorFlow, TensorRT, PyTorch, ONNX, XGBoost, and other frameworks. section 7.4.3
• GPU throughput and utilization improvements through concurrent execution and dynamic batching. section 7.4.3
• A model-server approach that reduces development and maintenance effort across multiple model types. section 7.7

Integrating Triton behind an existing prediction API

Keep the public prediction interface and cloud model storage unchanged by placing Triton Kubernetes pods behind the existing API. Give the pods a shared model repository, then use a sidecar to download models from cloud storage, register them with Triton, and translate incoming requests into Triton API calls. section 7.4.5

Figure 7.10 Triton Inference Server high-level architecture (Source: NVIDIA Developer, https://developer.nvidia.com/nvidia-triton-inference-server)

Use the architecture diagram to trace requests from REST or gRPC entry points through model schedulers and backends. section 7.4.3

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Defines protocols, repositories, schedulers, backends, batching, and deployment modes without detailing backend implementations.

Examples

Uses model-repository configurations, TorchScript export, and a Kubernetes sidecar integration pattern.

What it assumes

Assumes familiarity with deployed models and prediction services while introducing model repositories, schedulers, and serving backends.

What you can do after

Operate/configure a multi-framework inference server and integrate it behind an existing prediction API.

How it connects

Connects model serving with REST, gRPC, Kubernetes, model repositories, dynamic batching, model ensembles, and framework-specific backends.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning System DesignNVIDIA Triton Inference Server is open-source serving software for deploying and executing machine learning models through standardized inference interfaces.

Compare how 2 books explain NVIDIA Triton Inference Service →

Related concepts

• Model Servingkind of · A model-serving platform is software infrastructure that hosts trained models and exposes inference by accepting requests and returning model outputs.
• language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• model repositoryuses · A model repository is a storage system containing machine-learning models and associated versions, artifacts, metadata, or configuration.
• ONNXuses · ONNX is an open standard for representing machine-learning models in a portable format that supports exchange and execution across frameworks and runtimes.
• sidecar processused by · The sidecar pattern deploys an auxiliary process or component alongside a primary application to provide supporting functionality separately from the application’s core code.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.