Profiling collects runtime measurements from model execution to expose bottlenecks and guide optimization. In ONNX Runtime, profiling is disabled by default; enabling it produces JSON data covering threading and per-operator latency. section 10.0 section 10.1
A practical profiling workflow is:
- Enable profiling on the inference session and run representative inference workloads. section 10.1
- Call end_profiling to retrieve the generated JSON file, whose records identify operators, execution providers, durations, shapes, and thread information. section 10.1
- Transform raw records into tables and aggregate total duration, invocation count, and percentage of inference time by operator. section 10.2
- Use Ort_perf_view or custom visualizations to locate expensive operators and prioritize model or runtime changes against latency, throughput, and memory KPIs. section 10.1 section 10.2
From regression model to operator report
A scikit-learn linear regression model converted to ONNX can be profiled over a test set. Processing the resulting JSON with mlprodict and plotting it with Plotly reveals which operators consume the most total and proportional inference time; the same process applies to larger ONNX models and LLMs. section 10.2
import onnxruntime as rt
sess_options = rt.SessionOptions()
sess_options.enable_profiling = True
Shows the minimal ONNX Runtime session configuration needed to enable runtime profiling. section 10.1
