ONNX

As explained in Machine Learning System Design

ONNX is an open format and interoperability standard for representing machine learning models across frameworks and runtimes.

ONNX provides a common model representation between training frameworks and production inference systems. It helps teams preserve research flexibility while deploying models across platforms, though direct framework serving may be more flexible and highly custom engines may be faster. section 15.3.1

Use ONNX as a practical deployment layer when you need to:

• Convert trained models from supported training frameworks into a static graph representation. section 10.6.2 section 15.3.1
• Run models with ONNX Runtime across platforms and hardware backends. section 15.3.1
• Connect optimized backends such as OpenVINO for CPUs or TensorRT for CUDA inference. section 15.3.1
• Pair model files with inference parameters, preprocessing choices, and reproducible training artifacts. section 10.6.2

Two-encoder training pipeline

A multimodal training pipeline can export both a text encoder and an image encoder to ONNX, save them with prompt-generation, image-preprocessing, and distance-function parameters, and record training metrics alongside the artifacts. section 10.6.2

Figure 15.3 A wide spectrum of tools with their focus on either research or production serving goals

Use the framework spectrum to compare research-oriented tools, universal formats, runtimes, and production-serving systems before choosing an inference path. section 15.3.1

How Machine Learning System Design covers it

How deep it goes

Explains conversion, runtime support, optimization tools, and deployment tradeoffs without detailing the graph specification.

Examples

Uses paired text and image encoders, static graph exports, inference parameters, metrics reports, and S3-stored artifacts.

What it assumes

Assumes familiarity with training frameworks, inference frameworks, model conversion, and target hardware backends.

What you can do after

Operate cross-framework inference through ONNX Runtime or hardware-specific backends, while weighing flexibility against peak performance.

How it connects

Connects training frameworks, model conversion, ONNX Runtime, OpenVINO, TensorRT, TVM, and production inference optimization.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Domain-Specific Small Language ModelsA serialized machine-learning model represented in the ONNX format for portable execution outside the training framework.
• 

  liveBookmanning.com
  Server-Side WebAssemblyONNX is an open model representation and interoperability standard for exchanging machine-learning models between frameworks and runtimes.
• 

  liveBookmanning.com
  LLMs in ProductionONNX is an open model format and ecosystem for representing neural-network models across different frameworks, tools, and hardware environments.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionAn open format and runtime ecosystem for representing and executing trained machine-learning models outside the original training framework.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionONNX is an open format and ecosystem for representing and exchanging machine-learning models across frameworks and runtimes.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA vendor-neutral standard and file format for representing neural network computations so models can be executed in different runtimes and devices.

Compare how 7 books explain ONNX →

Related concepts

• ONNX Runtimeused by · ONNX Runtime is an open-source, cross-platform inference engine for executing and optimizing machine-learning models represented in the ONNX format.
• model exportproduced by · Model export is the process of converting a trained machine-learning model into a portable representation suitable for another format, framework, runtime, or deployment environment.
• TensorFlow Servingalternative to · TensorFlow Serving is an open-source system for deploying trained machine-learning models and exposing them through production inference APIs.
• artifact packagingused by · Artifact packaging is the preparation of a trained model and its associated files and requirements as a deployable unit.
• Optimumproduced by · Optimum is a Hugging Face library for optimizing and deploying machine-learning models across inference engines and hardware platforms.
• language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.