How deeply does Domain-Specific Small Language Models cover execution provider?

Explains node assignment, provider ordering, provider-specific optimization, data transfers, and profiling, but defers backend implementation internals.

Execution provider

As explained in Domain-Specific Small Language Models

An ONNX Runtime backend that assigns graph nodes to a specific hardware/runtime implementation such as CPU, CUDA, or ROCm.

An execution provider is an ONNX Runtime backend that assigns graph nodes or subgraphs to hardware-specific kernels, drivers, and libraries. It lets one ONNX model target CPUs, GPUs, edge devices, or specialized accelerators while using the deployment platform’s compute capabilities. section 5.4

Common provider choices map the model to different hardware and runtime stacks:

CPUExecutionProvider supplies a vendor-agnostic CPU path. section 5.4
CUDAExecutionProvider accelerates computation on NVIDIA GPUs; TensorRT provides another supported NVIDIA GPU backend. section 5.4
ROCm and MIGraphX target AMD GPUs, while DirectML targets Windows GPUs. section 5.4
OpenVINO, oneDNN, XNNPACK, Android NNAPI, and Qualcomm QNN address other CPU, mobile, edge, and chipset environments; TVM is listed as preview. section 5.4

Provider selection is an operational deployment decision rather than a model-format change. A session can prioritize a GPU provider and include CPU as a fallback. Graph optimizations are provider-aware: extended fusions apply after partitioning to CPU, CUDA, or ROCm-assigned nodes, while layout optimization applies to CPU-assigned nodes. section 5.4 section 5.6.1

CUDA with CPU fallback

Configure an inference session with providers = ['CUDAExecutionProvider', 'CPUExecutionProvider']; ORT attempts the CUDA path first and retains a CPU path for supported fallback behavior. Inspect available backends with rt.get_available_providers().

GPU execution can still lose time to data movement. ORT places ordinary inputs and outputs on the CPU by default, so IOBinding can keep tensors on the execution device and avoid unnecessary CPU–GPU copies. section 5.6.2 section 5.7

Provider choice also affects diagnosis: ORT profiling records per-operator latency and the responsible execution provider, and Ort_perf_view summarizes those measurements for performance analysis. section 10.1

How Domain-Specific Small Language Models covers it

Cover of Domain-Specific Small Language Models

liveBook manning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

ONNX Runtimeuses · ONNX Runtime is an open-source, cross-platform inference engine for executing and optimizing machine-learning models represented in the ONNX format.
GPU kerneluses · A GPU kernel is an executable program or function that performs computation on a graphics processing unit, commonly using parallel threads.
ONNXuses · ONNX is an open standard for representing machine-learning models in a portable format that supports exchange and execution across frameworks and runtimes.
environmentuses · An environment is the external context, including its entities and conditions, in which a system, process, or agent exists or operates.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.