Execution provider
An ONNX Runtime execution provider is a backend that assigns and executes model-graph operations on a particular hardware or software target.
Domain-Specific Small Language Models
An ONNX Runtime backend that assigns graph nodes to a specific hardware/runtime implementation such as CPU, CUDA, or ROCm.
- How it is used
- Assigns ONNX graph nodes or subgraphs to deployment-specific implementations that use available hardware, drivers, and libraries.
- How deep it goes
- Defines node assignment, surveys provider options and maturity, and shows session configuration; does not detail provider implementation internals.
- Examples
- Shows a CUDA-before-CPU inference session and a CPU-backed linear regression model with provider-labeled profiling data.
- What you can do after
- Operate/configure it by selecting providers for an ORT inference session, then inspect provider-level operator performance with profiling.
Read how Domain-Specific Small Language Models explains execution provider →
Quantization and Fast Inference
An execution provider is an ONNX Runtime backend that assigns and executes model-graph operations on a particular hardware or software target.
- How it is used
- Selects which runtime kernels dispatch for model operations on the target hardware.
- How deep it goes
- Explains provider selection as the kernel-dispatch stage and identifies silent FP32 fallback when target-dtype kernels are unavailable.
- Examples
- Names CUDA, TensorRT, OpenVINO, DirectML, ROCm, CPU, and TensorRT and OpenVINO provider configurations.
- What you can do after
- Diagnose whether target-dtype kernels actually dispatch by verifying execution dtype on the deployment host.
Read how Quantization and Fast Inference explains execution provider →
What to know first
- CPUA central processing unit (CPU) is a computer processor that executes program instructions and performs general-purpose computational operations.
- ONNXONNX (Open Neural Network Exchange) is an open standard and format for representing and exchanging machine-learning models across frameworks, tools, and runtimes.
Related concepts
- ONNX Runtimeuses · ONNX Runtime is an open-source, cross-platform runtime for executing and optimizing machine-learning models in the ONNX format.
- ONNXuses · ONNX (Open Neural Network Exchange) is an open standard and format for representing and exchanging machine-learning models across frameworks, tools, and runtimes.
- environmentuses · An environment is the external context, system, or conditions within which an agent or application operates and interacts.
- runtime contractconfigured by · A runtime contract is a set of expectations or constraints governing a component’s behavior and compatibility during execution.
Questions about execution provider
What is execution provider?
An ONNX Runtime execution provider is a backend that assigns and executes model-graph operations on a particular hardware or software target.
Which Manning books explain execution provider?
Domain-Specific Small Language Models and Quantization and Fast Inference.
What should I know before learning execution provider?
Manning books that teach execution provider usually introduce CPU and ONNX first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

