Chapter 5: Exploring ONNX

Introduction

Introduces ONNX as a framework relevant to model optimization, quantization, and portability.

• ONNX benefits for models: Explain the roles the chapter attributes to ONNX: model optimization, quantization, and portability across frameworks and hardware vendors.
  Why it matters: This gives you the reason ONNX is relevant before the chapter explains its components.

Key terms: ONNX framework, model optimization, quantization, portability

5.1 The ONNX format

This section explains why ONNX makes trained models portable, how its format represents models, and what can be done after conversion.

• Framework and hardware portability: Explain how ONNX reduces dependence on the framework used to create a model and on a particular hardware accelerator.
  Why it matters: Portability lets you retain useful models even when your preferred software or hardware changes.
• ONNX graph representation: Describe how an ONNX file represents a trained model using operators and typed tensors.
  Why it matters: The graph is the portable description that another supported platform can use to understand the model.
• Converted model options: Trace the options the section gives after converting a trained model to ONNX, including how the resulting model can be run.
  Why it matters: Conversion provides several deployment paths rather than requiring one way to run a model.
• Format and runtime distinction: Distinguish the ONNX model format from runtime support, and explain the limited purpose of the Python runtime described in the section.
  Why it matters: Separating the model description from its execution tool helps you choose an appropriate runtime.

Key terms: ONNX, interoperability, portability, lock-in, ONNX graph, operators, typed tensor, custom operators, convert to ONNX, ONNX Runtime

5.2 ONNX operators and types

This section explains ONNX operators and tensor types, then uses a linear regression model to show how an ONNX graph is built, checked, serialized, and inspected.

• Operators and tensor types: Explain how operators and typed tensors form the building blocks of an ONNX model.
  Why it matters: These are the pieces used to express the example model as an ONNX graph.
• Linear regression graph construction: Trace how the section represents Y = XA + B as an ONNX graph using typed variables, operator nodes, a graph, and a model.
  Why it matters: The worked example shows how a mathematical expression becomes an ONNX model.
• Model validation and inspection: Describe how the created ONNX model can be checked, serialized, inspected, and restored.
  Why it matters: Checking and inspecting the graph helps confirm what was built before using or saving it.

Key terms: operators, tensors, element type, shape, MatMul, Add, make_tensor_value_info, make_node, make_graph, make_model

5.3 The ONNX runtime

This section explains what ONNX Runtime does, the basic sequence for running ONNX inference, and a browser-based serving option.

• ONNX Runtime’s execution role: Explain how ONNX Runtime differs from the ONNX format and why it is commonly used for inference across deployment environments.
  Why it matters: Distinguishing the model format from the runtime clarifies what actually executes a converted model.
• ORT inference command sequence: Trace the section’s fixed sequence for running inference with an ONNX model, including how to identify the required input name.
  Why it matters: The same inference sequence applies even when a model’s input and output handling changes.
• ONNX Runtime Web option: Describe ONNX Runtime Web as an alternative way to serve an ONNX model and state the benefits identified in the section.
  Why it matters: It shows that an optimized ONNX model can also be served through a web-based runtime.

Key terms: ONNX format, ONNX Runtime, ORT, hardware acceleration, inference, InferenceSession, get_inputs, get_outputs, input preprocessing, output postprocessing

5.4 ONNX runtime providers

This section explains how ONNX Runtime execution providers use deployment hardware, how providers differ in support and maturity, and how to inspect and prioritize those available.

• Execution provider role: Explain how ONNX Runtime uses execution providers to run an ONNX model on the hardware and software available in a deployment environment.
  Why it matters: Providers connect a model’s execution to the capabilities of the machine where it runs.
• Provider compatibility and maturity: Distinguish providers by their supported platforms and release status, and explain the caution needed for preview providers.
  Why it matters: A provider must suit the target environment, and its release status affects deployment confidence.
• Available provider preference: Describe how to check which providers are available in a destination environment and how to express a preferred order when more than one is available.
  Why it matters: Checking availability and setting an order lets deployment express a hardware preference without assuming that hardware is present.

Key terms: ORT, execution providers, nodes, subgraphs, stable execution providers, preview, NVIDIA CUDA, available providers, Python list, CUDA provider

5.5 ONNX for LLMs on CPU

This section walks through exporting a BERT model to ONNX, validating it, and running and benchmarking CPU inference.

• BERT to ONNX export: Describe how the section converts bert-base-uncased to ONNX for a CPU-only setup, including the export settings that allow variable-sized inputs.
  Why it matters: These export choices produce the ONNX model used in the section’s validation and CPU inference examples.
• Exported model validation: Distinguish checking the exported model’s schema from checking its behavior against the original model.
  Why it matters: A valid schema alone does not show that the converted model produces comparable outputs.
• CPU inference inputs: Describe how the exported model is run through an ONNX Runtime session and how its inputs differ from those of the original model.
  Why it matters: The ONNX Runtime example needs inputs in the form and names the exported model expects.
• CPU benchmark and optimization: State what the example reports when benchmarking the original and ONNX-converted models, and what further step it takes with the converted model.
  Why it matters: The benchmark gives a concrete CPU-inference comparison while the optimization step shows what the example tries next.

Key terms: bert-base-uncased, PyTorch’s ONNX export function, constant folding, variable-length axes, check_model, allclose, rtol, atol, ONNX Runtime session, NumPy arrays

5.6 ONNX for LLMs on GPU

This section explains the changes needed to convert and run an ONNX model on an NVIDIA GPU rather than a CPU.

• GPU ONNX inference setup: Explain that running an ONNX-converted model on a GPU requires an NVIDIA GPU with CUDA drivers, the GPU version of ONNX Runtime, and an inference session configured to use the CUDA execution provider.
  Why it matters: Choosing the CUDA execution provider is the key runtime change for GPU inference.
• GPU accelerated model conversion: Describe how the example loads the Hugging Face model into GPU memory to use hardware acceleration during conversion to ONNX, while otherwise following the CPU conversion approach.
  Why it matters: This separates accelerating conversion from configuring GPU inference.

Key terms: ONNX inference session, CUDA execution provider, onnxruntime for GPU, GPU memory, hardware acceleration, convert the model to ONNX

5.6.1 ONNX for GPT on GPU

This section exports GPT-2 to ONNX for GPU inference and explains how to optimize the exported graph offline or when an inference session starts.

• GPT-2 export and inference: Describe the GPT-2 workflow: download the model, prepare input without an unused attention mask, convert it to ONNX, then create an inference session with the CUDA provider and call run. Distinguish using the GPU for conversion from requiring it for conversion.
  Why it matters: It separates the choice of where to convert the model from the choice of where to run inference.
• Transformer graph optimization options: Explain why an exported ONNX model may need additional optimization and describe the optimizer inputs and optimization levels presented for the GPT-2 example.
  Why it matters: It gives a programmatic way to improve a converted graph without assuming export has already done everything possible.
• Offline versus online optimization: Distinguish optimizing and saving an ONNX model offline from enabling graph optimization when an inference session starts, including the startup-time tradeoff.
  Why it matters: Choosing when to optimize affects application startup time, especially for complex models.

Key terms: GPT-2 small, return_attention_mask, inference session, CUDA runtime provider, ONNX Runtime, Transformer optimizer, optimize_model, graph optimization level, constant folding, graph partitioning

5.6.2 I/O binding

This section explains why ONNX Runtime I/O binding can reduce CPU–GPU data-transfer overhead and outlines how to bind inputs and outputs for GPU inference.

• CPU–GPU transfer overhead: Explain why ONNX Runtime’s default data placement can add overhead when an inference session runs on a GPU, and how IOBinding addresses it.
  Why it matters: It explains when I/O binding is useful rather than treating it as a required step for every inference session.
• Input and output binding steps: Describe the example’s IOBinding workflow, including how running with GPU-bound inputs differs from also binding the output.
  Why it matters: Binding inputs alone does not keep the result on the GPU; the output needs its own binding.

Key terms: ONNX Runtime, ORT, IOBinding, execution providers, run_with_iobinding, CUDA, input, output

5.7 Summary