ONNX

As explained in Domain-Specific Small Language Models

A serialized machine-learning model represented in the ONNX format for portable execution outside the training framework.

ONNX (Open Neural Network Exchange) is an open standard for serializing machine-learning models as portable computation graphs. It reduces dependence on a particular training framework or hardware accelerator, allowing models from frameworks such as PyTorch, TensorFlow, Keras, scikit-learn, and XGBoost to move toward different runtimes and platforms. section 5.1 section 5.7

The format and ONNX Runtime are related but distinct. The format describes the model graph; ONNX Runtime executes it through hardware-specific execution providers such as CPU and CUDA. Models can be exported, validated, optimized, quantized, and then run in a supported runtime. section 5.1 section 5.6.2

An ONNX model is built from:

• Operators such as MatMul, Add, reductions, tensor transformations, neural-network layers, and activation functions. section 5.2
• Strongly typed tensors described by element type, shape, and values. section 5.2
• A serialized graph using Protocol Buffers, with model metadata and inputs and outputs. section 5.2

Linear regression graph

The expression Y = XA + B becomes a graph with a MatMul node followed by an Add node. Python bindings declare tensor information, create nodes and a graph, assemble the model, check it with check_model, and serialize it to a .onnx file. section 5.2

A practical deployment path exports a trained model, validates schema and numerical agreement with the original implementation, selects an execution provider, and measures performance. Runtime optimization can apply constant folding, redundant-node removal, operator fusion, and layout transformations. On GPU systems, IOBinding keeps inputs or outputs on the target device to avoid unnecessary CPU–GPU copies. section 5.5 section 5.6.2

Transformer deployment

A BERT question-answering model can be exported from PyTorch with dynamic batch and sequence axes, checked, compared against the original outputs, and executed with ONNX Runtime. The same pattern supports CUDA execution and transformer-specific optimization; ONNX-based quantization can further produce an 8-bit model. section 5.5 section 5.6 section 6.3

Figure 5.1 ONNX overview

Use the ecosystem overview to trace how framework models move through the ONNX format toward runtimes, conversion, optimization, and deployment targets. section 5.1

How Domain-Specific Small Language Models covers it

How deep it goes

Defines graphs, operators, tensors, serialization, runtimes, validation, optimization, and IOBinding, then applies them to transformer deployment.

Examples

Demonstrates linear regression graphs, BERT question answering, GPT-2 generation, CUDA execution, IOBinding, and 8-bit classification.

What it assumes

Assumes familiarity with trained ML models and inference frameworks while teaching ONNX graphs, operators, tensor typing, export, and runtime execution.

What you can do after

Build and operate portable model graphs with ONNX Runtime, validate outputs, select CPU or CUDA execution, optimize graphs, and quantize models.

How it connects

Connects model export with ONNX Runtime, execution providers, Protocol Buffers, quantization, graph optimization, Transformers, and Optimum.

liveBookmanning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Server-Side WebAssemblyONNX is an open model representation and interoperability standard for exchanging machine-learning models between frameworks and runtimes.
• 

  liveBookmanning.com
  LLMs in ProductionONNX is an open model format and ecosystem for representing neural-network models across different frameworks, tools, and hardware environments.
• 

  liveBookmanning.com
  Machine Learning System DesignONNX is an open format and interoperability standard for representing machine learning models across frameworks and runtimes.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionAn open format and runtime ecosystem for representing and executing trained machine-learning models outside the original training framework.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionONNX is an open format and ecosystem for representing and exchanging machine-learning models across frameworks and runtimes.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA vendor-neutral standard and file format for representing neural network computations so models can be executed in different runtimes and devices.

Compare how 7 books explain ONNX →

Related concepts

• ONNX Runtimeused by · ONNX Runtime is an open-source, cross-platform inference engine for executing and optimizing machine-learning models represented in the ONNX format.
• model exportproduced by · Model export is the process of converting a trained machine-learning model into a portable representation suitable for another format, framework, runtime, or deployment environment.
• TensorFlow Servingalternative to · TensorFlow Serving is an open-source system for deploying trained machine-learning models and exposing them through production inference APIs.
• artifact packagingused by · Artifact packaging is the preparation of a trained model and its associated files and requirements as a deployable unit.
• Optimumproduced by · Optimum is a Hugging Face library for optimizing and deploying machine-learning models across inference engines and hardware platforms.
• language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.