ONNX Runtime

As explained in Machine Learning System Design

ONNX Runtime is a cross-platform runtime for executing ONNX machine-learning models with hardware-specific optimizations.

ONNX Runtime executes ONNX models across platforms, using hardware-specific optimizations to balance deployment flexibility with production performance. section 15.3.1

Use ONNX Runtime when:

• Training and serving use different frameworks, making an interoperable model format valuable. section 15.3.1
• One system must target multiple platforms or hardware backends. section 15.3.1
• You want more performance than directly serving flexible research frameworks, without building a fully custom solution. section 15.3.1

A practical deployment choice

A team can convert models from supported training frameworks to ONNX, then use ONNX Runtime for inference while selecting suitable backends such as OpenVINO for Intel CPUs or TensorRT for CUDA. section 15.3.1

Figure 15.3 A wide spectrum of tools with their focus on either research or production serving goals

Compare ONNX Runtime with tools positioned closer to research flexibility or production-serving specialization. section 15.3.1

How Machine Learning System Design covers it

How deep it goes

Explains its ecosystem role, backend support, and tradeoffs, but stops short of implementation details or benchmarking procedures.

Examples

Uses ONNX model conversion and OpenVINO or TensorRT backends as deployment examples.

What you can do after

Evaluate inference-framework choices against portability, hardware optimization, flexibility, and performance requirements.

How it connects

Connects ONNX Runtime with model conversion, inference frameworks, OpenVINO, TensorRT, TVM, AITemplate, CoreML, and TensorFlow Lite.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Domain-Specific Small Language ModelsA runtime library that efficiently executes ONNX models across hardware targets such as CPUs, GPUs, and edge devices.
• 

  liveBookmanning.com
  LLMs in ProductionONNX Runtime is a cross-platform inference engine that optimizes and executes models represented in ONNX format.

Compare how 3 books explain ONNX Runtime →

Related concepts

• ONNXuses · ONNX is an open standard for representing machine-learning models in a portable format that supports exchange and execution across frameworks and runtimes.
• execution providerused by · An execution provider is an ONNX Runtime backend that assigns and executes model-graph operations on a specific hardware or software target.
• TensorRTuses · TensorRT is NVIDIA’s software development kit and inference runtime for optimizing and executing machine-learning models on NVIDIA hardware.
• profilinguses · Profiling is the measurement and analysis of a program’s runtime behavior and resource use to characterize performance and identify bottlenecks.
• generationuses · Generation is the computational production of outputs from supplied inputs or context, often by applying a model to produce predictions or new content.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.