liveBook · Manning

ONNX Runtime

ONNX Runtime is an open-source, cross-platform runtime for executing and optimizing machine-learning models in the ONNX format.

How 5 Manning books explain it

Domain-Specific Small Language Models

A runtime library that efficiently executes ONNX models across hardware targets such as CPUs, GPUs, and edge devices.

How it is used
Runs portable ONNX models on target hardware for inference after conversion from a training framework.
How deep it goes
Works through session setup, validation, optimization, I/O binding, and profiling, without descending into operator-kernel implementation.
Examples
Uses BERT question answering on CPU, GPT-2 on GPU, tensor addition for I/O binding, and linear regression for profiling.
What you can do after
Apply it via ONNX Runtime sessions to run and validate exported models, then configure device execution and investigate performance.

Read how Domain-Specific Small Language Models explains ONNX Runtime →

LLMs in Production

ONNX Runtime is a cross-platform inference engine that optimizes and executes models represented in ONNX format.

How it is used
Provides a portable inference path when TensorRT compatibility problems make model compilation difficult.
How deep it goes
Defines ONNX Runtime’s role and deployment rationale but stops short of explaining its internal optimization algorithms.
Examples
Uses a Hugging Face Optimum CLI command to export and optimize WizardCoder in ONNX format.
What you can do after
Apply it via Optimum to export models into ONNX for portable inference deployment.

Read how LLMs in Production explains ONNX Runtime →

Quantization and Fast Inference

ONNX Runtime is an inference engine for executing models represented in the Open Neural Network Exchange format.

How it is used
Bridges framework-trained models to deployment by executing portable ONNX graphs and applying runtime-side quantization.
How deep it goes
Distinguishes ONNX portability from ORT execution, then demonstrates export and quantization without detailing ORT internals.
Examples
Uses ResNet-18 export and calibration, and compares ResNet-18 and BERT-base quantization results.
What you can do after
Apply dynamic or calibrated static quantization to exported ONNX models, weighing operator coverage against compression and accuracy.

Read how Quantization and Fast Inference explains ONNX Runtime →

Machine Learning System Design

ONNX Runtime is a cross-platform runtime for executing ONNX machine-learning models with hardware-specific optimizations.

How it is used
Positions ONNX Runtime as a cross-platform inference engine for balancing framework flexibility and production performance.
How deep it goes
Explains its ecosystem role, backend support, and tradeoffs, but stops short of implementation details or benchmarking procedures.
Examples
Uses ONNX model conversion and OpenVINO or TensorRT backends as deployment examples.
What you can do after
Evaluate inference-framework choices against portability, hardware optimization, flexibility, and performance requirements.

Read how Machine Learning System Design explains ONNX Runtime →

Fight Fraud with Machine Learning

ONNX Runtime is an open-source cross-platform inference engine for executing and optimizing machine-learning models represented in the Open Neural Network Exchange format.

How it is used
Optimizes deep-learning model inference for deployment on available hardware.
How deep it goes
Defines its optimization role and cross-framework context but stops short of implementation commands or runtime configuration.
Examples
Uses a PyTorch-to-ONNX conversion pipeline and compares ONNX Runtime with OpenVINO.
What you can do after
Apply it via ONNX Runtime to optimize inference for converted deep-learning models.

Read how Fight Fraud with Machine Learning explains ONNX Runtime →

What to know first

Related concepts

Questions about ONNX Runtime

What is ONNX Runtime?

ONNX Runtime is an open-source, cross-platform runtime for executing and optimizing machine-learning models in the ONNX format.

Which Manning books explain ONNX Runtime?

Domain-Specific Small Language Models, LLMs in Production, Quantization and Fast Inference, Machine Learning System Design and Fight Fraud with Machine Learning.

What should I know before learning ONNX Runtime?

Manning books that teach ONNX Runtime usually introduce CPU, language model, quantization and Python first.

What is the difference between ONNX Runtime and TensorRT?

ONNX Runtime is an open-source, cross-platform runtime for executing and optimizing machine-learning models in the ONNX format. TensorRT is NVIDIA’s software development kit for optimizing and executing machine-learning inference on NVIDIA hardware.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.