Triton backend

As explained in Designing Deep Learning Systems: A guide for software engineers

An implementation that executes a model of a particular framework or model type within Triton.

A Triton backend executes a model of a particular framework or model type. Triton’s per-model scheduler optionally batches requests, then sends them to the matching backend for inference; the backend returns the requested outputs. section 7.4.3

A backend fits into Triton’s serving architecture as follows:

• Receives batched inference inputs from a model scheduler. section 7.4.3
• Wraps a framework or runtime such as PyTorch, TensorFlow, TensorRT, or ONNX Runtime. section 7.4.3
• Extends Triton through the backend C API with custom preprocessing, postprocessing, or new framework support. section 7.4.3

Figure 7.10 Triton Inference Server high-level architecture (Source: NVIDIA Developer, https://developer.nvidia.com/nvidia-triton-inference-server)

Use the architecture diagram to trace requests from REST or gRPC entry points through model scheduling and backend execution. section 7.4.3

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Defines scheduler handoff, framework wrapping, and C API extension, but does not detail backend implementation internals.

Examples

Uses PyTorch, TensorFlow, TensorRT, and ONNX Runtime backends as concrete framework examples.

What you can do after

Apply it via Triton’s backend C API to add custom processing or support a new deep learning framework.

How it connects

Connects model repositories, model schedulers, dynamic batching, inference runtimes, and model management APIs.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.