ONNX Runtime
ONNX Runtime is an open-source, cross-platform inference engine for executing and optimizing machine-learning models represented in the ONNX format.
Domain-Specific Small Language Models
A runtime library that efficiently executes ONNX models across hardware targets such as CPUs, GPUs, and edge devices.
- How it is used
- Deploys portable ONNX models for efficient inference across CPUs, GPUs, and other hardware targets.
- How deep it goes
- Explains sessions, providers, graph optimization, IOBinding, validation, and profiling with runnable LLM and tensor examples.
- Examples
- Uses BERT question answering, GPT-2 generation, a tensor-addition model, SQuAD, and ONNX Runtime performance profiles.
- What you can do after
- Operate/configure ONNX Runtime sessions, select providers, optimize graphs, bind device memory, validate outputs, and profile inference.
Read how Domain-Specific Small Language Models explains ONNX Runtime →
LLMs in Production
ONNX Runtime is a cross-platform inference engine that optimizes and executes models represented in ONNX format.
- How it is used
- Provides a portable inference path when TensorRT compatibility problems make model compilation difficult.
- How deep it goes
- Defines ONNX Runtime’s role and deployment rationale but stops short of explaining its internal optimization algorithms.
- Examples
- Uses a Hugging Face Optimum CLI command to export and optimize WizardCoder in ONNX format.
- What you can do after
- Apply it via Optimum to export models into ONNX for portable inference deployment.
Machine Learning System Design
ONNX Runtime is a cross-platform runtime for executing ONNX machine-learning models with hardware-specific optimizations.
- How it is used
- Positions ONNX Runtime as a cross-platform inference engine for balancing framework flexibility and production performance.
- How deep it goes
- Explains its ecosystem role, backend support, and tradeoffs, but stops short of implementation details or benchmarking procedures.
- Examples
- Uses ONNX model conversion and OpenVINO or TensorRT backends as deployment examples.
- What you can do after
- Evaluate inference-framework choices against portability, hardware optimization, flexibility, and performance requirements.
Read how Machine Learning System Design explains ONNX Runtime →
Related concepts
- ONNXuses · ONNX is an open standard for representing machine-learning models in a portable format that supports exchange and execution across frameworks and runtimes.
- execution providerused by · An execution provider is an ONNX Runtime backend that assigns and executes model-graph operations on a specific hardware or software target.
- TensorRTuses · TensorRT is NVIDIA’s software development kit and inference runtime for optimizing and executing machine-learning models on NVIDIA hardware.
- profilinguses · Profiling is the measurement and analysis of a program’s runtime behavior and resource use to characterize performance and identify bottlenecks.
- generationuses · Generation is the computational production of outputs from supplied inputs or context, often by applying a model to produce predictions or new content.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.