NVIDIA Triton Inference Service
NVIDIA Triton Inference Server is open-source software for deploying and serving machine-learning models through standardized interfaces across frameworks and hardware backends.
Designing Deep Learning Systems: A guide for software engineers
NVIDIA Triton Inference Server is an open-source inference-serving system that supports models from multiple machine-learning frameworks and hardware backends.
- How it is used
- Deploys heterogeneous machine-learning models as a shared inference platform for cloud, data-center, and edge workloads.
- How deep it goes
- Defines protocols, repositories, schedulers, backends, batching, and deployment modes without detailing backend implementations.
- Examples
- Uses model-repository configurations, TorchScript export, and a Kubernetes sidecar integration pattern.
- What you can do after
- Operate/configure a multi-framework inference server and integrate it behind an existing prediction API.
Machine Learning System Design
NVIDIA Triton Inference Server is open-source serving software for deploying and executing machine learning models through standardized inference interfaces.
- How it is used
- Serves ONNX models for real-time inference through HTTP while supporting a shared model strategy with separate batch operations.
- How deep it goes
- Explains framework selection, HTTP integration, model reuse, and latency configuration without detailing Triton internals.
- Examples
- Demonstrates PhotoStock search serving with two-average and 100-peak queries per second.
- What you can do after
- Operate Triton for HTTP-based real-time model serving with latency-aware batching and a separate scalable batch path.
Read how Machine Learning System Design explains NVIDIA Triton Inference Service →
Related concepts
- Model Servingkind of · A model-serving platform is software infrastructure that hosts trained models and exposes inference by accepting requests and returning model outputs.
- language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- sidecar processused by · The sidecar pattern deploys an auxiliary process or component alongside a primary application to provide supporting functionality separately from the application’s core code.
- ONNXuses · ONNX is an open standard for representing machine-learning models in a portable format that supports exchange and execution across frameworks and runtimes.
- TensorFlow Servingalternative to · TensorFlow Serving is an open-source system for deploying trained machine-learning models and exposing them through production inference APIs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.