ONNX Runtime
ONNX Runtime is an open-source, cross-platform runtime for executing and optimizing machine-learning models in the ONNX format.
Domain-Specific Small Language Models
A runtime library that efficiently executes ONNX models across hardware targets such as CPUs, GPUs, and edge devices.
- How it is used
- Runs portable ONNX models on target hardware for inference after conversion from a training framework.
- How deep it goes
- Works through session setup, validation, optimization, I/O binding, and profiling, without descending into operator-kernel implementation.
- Examples
- Uses BERT question answering on CPU, GPT-2 on GPU, tensor addition for I/O binding, and linear regression for profiling.
- What you can do after
- Apply it via ONNX Runtime sessions to run and validate exported models, then configure device execution and investigate performance.
Read how Domain-Specific Small Language Models explains ONNX Runtime →
LLMs in Production
ONNX Runtime is a cross-platform inference engine that optimizes and executes models represented in ONNX format.
- How it is used
- Provides a portable inference path when TensorRT compatibility problems make model compilation difficult.
- How deep it goes
- Defines ONNX Runtime’s role and deployment rationale but stops short of explaining its internal optimization algorithms.
- Examples
- Uses a Hugging Face Optimum CLI command to export and optimize WizardCoder in ONNX format.
- What you can do after
- Apply it via Optimum to export models into ONNX for portable inference deployment.
Quantization and Fast Inference
ONNX Runtime is an inference engine for executing models represented in the Open Neural Network Exchange format.
- How it is used
- Bridges framework-trained models to deployment by executing portable ONNX graphs and applying runtime-side quantization.
- How deep it goes
- Distinguishes ONNX portability from ORT execution, then demonstrates export and quantization without detailing ORT internals.
- Examples
- Uses ResNet-18 export and calibration, and compares ResNet-18 and BERT-base quantization results.
- What you can do after
- Apply dynamic or calibrated static quantization to exported ONNX models, weighing operator coverage against compression and accuracy.
Read how Quantization and Fast Inference explains ONNX Runtime →
Machine Learning System Design
ONNX Runtime is a cross-platform runtime for executing ONNX machine-learning models with hardware-specific optimizations.
- How it is used
- Positions ONNX Runtime as a cross-platform inference engine for balancing framework flexibility and production performance.
- How deep it goes
- Explains its ecosystem role, backend support, and tradeoffs, but stops short of implementation details or benchmarking procedures.
- Examples
- Uses ONNX model conversion and OpenVINO or TensorRT backends as deployment examples.
- What you can do after
- Evaluate inference-framework choices against portability, hardware optimization, flexibility, and performance requirements.
Read how Machine Learning System Design explains ONNX Runtime →
Fight Fraud with Machine Learning
ONNX Runtime is an open-source cross-platform inference engine for executing and optimizing machine-learning models represented in the Open Neural Network Exchange format.
- How it is used
- Optimizes deep-learning model inference for deployment on available hardware.
- How deep it goes
- Defines its optimization role and cross-framework context but stops short of implementation commands or runtime configuration.
- Examples
- Uses a PyTorch-to-ONNX conversion pipeline and compares ONNX Runtime with OpenVINO.
- What you can do after
- Apply it via ONNX Runtime to optimize inference for converted deep-learning models.
Read how Fight Fraud with Machine Learning explains ONNX Runtime →
What to know first
- CPUA central processing unit (CPU) is a computer processor that executes program instructions and performs general-purpose computational operations.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- quantizationModel quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
- PythonPython is a high-level, general-purpose, dynamically typed programming language with readable syntax and support for multiple programming paradigms.
Related concepts
- ONNXuses · ONNX (Open Neural Network Exchange) is an open standard and format for representing and exchanging machine-learning models across frameworks, tools, and runtimes.
- execution providerused by · An ONNX Runtime execution provider is a backend that assigns and executes model-graph operations on a particular hardware or software target.
- TensorRTalternative to · TensorRT is NVIDIA’s software development kit for optimizing and executing machine-learning inference on NVIDIA hardware.
- ONNX dynamic quantizationused by · ONNX dynamic quantization is a model-quantization technique that quantizes weights ahead of inference and determines activation quantization parameters dynamically during execution.
- GGUFcontrasts with · GGUF is a binary file format for storing machine-learning model tensors and associated metadata, designed to support efficient model loading and inference.
Questions about ONNX Runtime
What is ONNX Runtime?
ONNX Runtime is an open-source, cross-platform runtime for executing and optimizing machine-learning models in the ONNX format.
Which Manning books explain ONNX Runtime?
Domain-Specific Small Language Models, LLMs in Production, Quantization and Fast Inference, Machine Learning System Design and Fight Fraud with Machine Learning.
What should I know before learning ONNX Runtime?
Manning books that teach ONNX Runtime usually introduce CPU, language model, quantization and Python first.
What is the difference between ONNX Runtime and TensorRT?
ONNX Runtime is an open-source, cross-platform runtime for executing and optimizing machine-learning models in the ONNX format. TensorRT is NVIDIA’s software development kit for optimizing and executing machine-learning inference on NVIDIA hardware.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




