liveBook · Manning

Execution provider

An ONNX Runtime execution provider is a backend that assigns and executes model-graph operations on a particular hardware or software target.

How 2 Manning books explain it

Domain-Specific Small Language Models

An ONNX Runtime backend that assigns graph nodes to a specific hardware/runtime implementation such as CPU, CUDA, or ROCm.

How it is used
Assigns ONNX graph nodes or subgraphs to deployment-specific implementations that use available hardware, drivers, and libraries.
How deep it goes
Defines node assignment, surveys provider options and maturity, and shows session configuration; does not detail provider implementation internals.
Examples
Shows a CUDA-before-CPU inference session and a CPU-backed linear regression model with provider-labeled profiling data.
What you can do after
Operate/configure it by selecting providers for an ORT inference session, then inspect provider-level operator performance with profiling.

Read how Domain-Specific Small Language Models explains execution provider →

Quantization and Fast Inference

An execution provider is an ONNX Runtime backend that assigns and executes model-graph operations on a particular hardware or software target.

How it is used
Selects which runtime kernels dispatch for model operations on the target hardware.
How deep it goes
Explains provider selection as the kernel-dispatch stage and identifies silent FP32 fallback when target-dtype kernels are unavailable.
Examples
Names CUDA, TensorRT, OpenVINO, DirectML, ROCm, CPU, and TensorRT and OpenVINO provider configurations.
What you can do after
Diagnose whether target-dtype kernels actually dispatch by verifying execution dtype on the deployment host.

Read how Quantization and Fast Inference explains execution provider →

What to know first

Related concepts

Questions about execution provider

What is execution provider?

An ONNX Runtime execution provider is a backend that assigns and executes model-graph operations on a particular hardware or software target.

Which Manning books explain execution provider?

Domain-Specific Small Language Models and Quantization and Fast Inference.

What should I know before learning execution provider?

Manning books that teach execution provider usually introduce CPU and ONNX first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.