InferenceSession is ONNX Runtime’s execution context for loading an ONNX model, selecting execution providers, preparing inputs, running inference, and retrieving outputs. The core sequence stays consistent across model architectures; preprocessing, postprocessing, and input/output handling vary. section 5.3
Use the session to:
- Load a model from a file or serialized model and choose CPU, CUDA, or another execution provider. section 5.6.1 section 10.1
- Discover valid input and output names with get_inputs and get_outputs, then pass correctly named arrays to run. section 5.3
- Configure graph optimization and enable profiling through SessionOptions when latency, throughput, or memory requires investigation. section 5.6.1 section 10.1
Running and profiling a model
Create an InferenceSession with a selected provider, call run with a mapping from input names to arrays, and call end_profiling to retrieve the generated JSON performance file. Ort_perf_view can summarize operator and provider timing. section 10.1 section 5.3
Serving optimized GPT-2
A FastAPI route can prepare tokenizer inputs, create a CPU InferenceSession for an optimized GPT-2 ONNX model, run generation, and return decoded text asynchronously. section 11.2.2
import onnxruntime as rt
onnx_model_path = "path_to_the_onnx_model"
session = rt.InferenceSession(onnx_model_path)
Shows the minimal session-construction pattern for loading an ONNX model before input discovery and inference. section 5.3
