How deeply does Machine Learning System Design cover NVIDIA Triton Inference Service?

Explains framework selection, HTTP integration, model reuse, and latency configuration without detailing Triton internals.

NVIDIA Triton Inference Service

As explained in Machine Learning System Design

NVIDIA Triton Inference Server is open-source serving software for deploying and executing machine learning models through standardized inference interfaces.

NVIDIA Triton Inference Server is an open-source, high-performance serving framework for deploying ONNX models behind standardized inference interfaces. It fits systems that need one model across batch and real-time paths while keeping their operational requirements separate. section 15.5.2

A practical deployment pattern separates workload concerns:

Expose real-time inference through Triton's HTTP API. section 15.5.2
Use a latency-optimized configuration for real-time traffic, such as keeping dynamic-batching queue delay below 10 milliseconds. section 15.5.2
Reuse the ONNX model for batch inference, but add input/output handling around the real-time inference service. section 15.5.2

PhotoStock search

A PhotoStock workload averages about two searches per second but can spike to roughly 100. Triton serves real-time requests from a pool of EC2-backed servers behind a load balancer, while SageMaker runs scalable batch jobs. section 15.5.2

How Machine Learning System Design covers it

liveBook manning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Designing Deep Learning Systems: A guide for software engineersNVIDIA Triton Inference Server is an open-source inference-serving system that supports models from multiple machine-learning frameworks and hardware backends.

Compare how 2 books explain NVIDIA Triton Inference Service →

Related concepts

Model Servingkind of · A model-serving platform is software infrastructure that hosts trained models and exposes inference by accepting requests and returning model outputs.
language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
sidecar processused by · The sidecar pattern deploys an auxiliary process or component alongside a primary application to provide supporting functionality separately from the application’s core code.
ONNXuses · ONNX is an open standard for representing machine-learning models in a portable format that supports exchange and execution across frameworks and runtimes.
TensorFlow Servingalternative to · TensorFlow Serving is an open-source system for deploying trained machine-learning models and exposing them through production inference APIs.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.