ONNX Runtime (ORT) is a runtime library for executing ONNX computation graphs across hardware targets. ONNX supplies the portable model format; ORT coordinates optimized operator kernels through execution providers such as CPU and CUDA, reducing dependence on a particular training framework or accelerator. section 5.1 section 5.7
Use ORT when deployment requires practical control over portability, performance, or device placement:
- Create an inference session with a selected execution provider, such as CPUExecutionProvider or CUDAExecutionProvider. section 5.5 section 5.6
- Validate an exported model by checking its ONNX schema and comparing outputs with the original framework using numerical tolerances. section 5.5
- Improve throughput or latency with graph optimization, including constant folding, redundant-node removal, operator fusion, and layout changes. section 5.6.1
- Use IOBinding when GPU inference would otherwise copy inputs or outputs between CPU and GPU memory. section 5.6.2
- Profile sessions during debugging to obtain JSON data for threading, operator latency, execution providers, and memory-related performance analysis. section 10.1
Running and optimizing BERT
Export bert-base-uncased from PyTorch to ONNX with dynamic batch and sequence axes, validate its outputs against PyTorch on SQuAD samples, and run it with the CPU provider. The measured average inference time falls from 455.49 ms in PyTorch CPU to 373.10 ms in ORT CPU, then to 355.48 ms after transformer optimization. The same model can use CUDAExecutionProvider on an NVIDIA GPU. section 5.5 section 5.6
session = onnxruntime.InferenceSession(export_model_path, sess_options,
➥providers=['CUDAExecutionProvider'])
Shows the minimal provider configuration needed to execute an ONNX-converted model on CUDA. section 5.6
