liveBook · Manning

Triton Inference Server

NVIDIA Triton Inference Server is open-source software for deploying and serving machine-learning models across multiple frameworks and hardware backends.

How 2 Manning books explain it

Designing Deep Learning Systems: A guide for software engineers

NVIDIA Triton Inference Server is an open-source inference-serving system that supports models from multiple machine-learning frameworks and hardware backends.

How it is used
Deploys heterogeneous machine-learning models as a shared inference platform for cloud, data-center, and edge workloads.
How deep it goes
Defines protocols, repositories, schedulers, backends, batching, and deployment modes without detailing backend implementations.
Examples
Uses model-repository configurations, TorchScript export, and a Kubernetes sidecar integration pattern.
What you can do after
Operate/configure a multi-framework inference server and integrate it behind an existing prediction API.

Read how Designing Deep Learning Systems: A guide for software engineers explains Triton Inference Server →

Machine Learning System Design

NVIDIA Triton Inference Server is open-source serving software for deploying and executing machine learning models through standardized inference interfaces.

How it is used
Serves ONNX models for real-time inference through HTTP while supporting a shared model strategy with separate batch operations.
How deep it goes
Explains framework selection, HTTP integration, model reuse, and latency configuration without detailing Triton internals.
Examples
Demonstrates PhotoStock search serving with two-average and 100-peak queries per second.
What you can do after
Operate Triton for HTTP-based real-time model serving with latency-aware batching and a separate scalable batch path.

Read how Machine Learning System Design explains Triton Inference Server →

Related concepts

Questions about Triton Inference Server

What is Triton Inference Server?

NVIDIA Triton Inference Server is open-source software for deploying and serving machine-learning models across multiple frameworks and hardware backends.

Which Manning books explain Triton Inference Server?

Designing Deep Learning Systems: A guide for software engineers and Machine Learning System Design.

What is the difference between Triton Inference Server and Triton?

NVIDIA Triton Inference Server is open-source software for deploying and serving machine-learning models across multiple frameworks and hardware backends. Triton is a programming language and compiler for writing and compiling high-performance GPU kernels.

What is Triton Inference Server a kind of?

Triton Inference Server is a kind of Model Serving. Model serving is the software infrastructure that hosts trained machine-learning models and handles inference requests, returning model outputs.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.