Chapter 10: Profiling insights
This introduction previews profiling ONNX-ported LLMs, turning profiling data into insights, and optimizing ONNX graphs.
After this chapter you will be able to
- ONNX Runtime profiling workflow. Describe how to enable profiling for an ONNX Runtime inference session, run inference, and obtain the resulting performance file.
- Operator profiling views and limits. Explain what the profiling JSON and Ort_perf_view show, including why the visualization becomes less useful for a large model.
- Profiling Data Transformation. Trace how the example moves from a profiled ONNX inference run to data suitable for analysis: save the raw JSON profiling data, load it, and use OnnxWholeSession to make a cleaner tabular representation.
- Operator Performance Views. Describe the three views built by aggregating profiling data by ONNX operator type: total duration, invocation count, and percentage of total inference time. Explain how they help identify opportunities for graph optimization.
- ONNX graph evaluation and inspection. Explain how ReferenceEvaluator helps inspect a full ONNX graph or an individual node, and describe what increasing its verbosity reveals.
- Custom operator fusion. Explain the section’s EyeLike-and-Add fusion example and how ReferenceEvaluator evaluates the resulting custom operator.
- GPT-2 offline graph optimization. Describe how the section applies the ORT optimizer to an ONNX-converted GPT-2 model and what graph and profiling changes it reports.
What it builds on
- The ONNX runtime (chapter 5) · Assumes familiarity with running inference through ONNX Runtime.
- ONNX operators and types (chapter 5) · Assumes familiarity with ONNX operators.
- The ONNX format (chapter 5) · The idea of representing a model in ONNX format.
- ONNX for GPT on GPU (chapter 5) · ORT optimization levels and offline graph optimization for ONNX LLMs.
Section by section
Introduction
This introduction previews profiling ONNX-ported LLMs, turning profiling data into insights, and optimizing ONNX graphs.
10.1 Profiling ONNX-ported LLMs
This section shows how to capture ONNX Runtime inference profiling data and use Ort_perf_view to inspect it, while noting the limits of its visualizations for large models.
- ONNX Runtime profiling workflow: Describe how to enable profiling for an ONNX Runtime inference session, run inference, and obtain the resulting performance file.
Why it matters: The profiling file provides measurements you can use to investigate whether inference meets performance targets. - Operator profiling views and limits: Explain what the profiling JSON and Ort_perf_view show, including why the visualization becomes less useful for a large model.
Why it matters: Reading the view correctly helps you distinguish useful performance summaries from a display that mainly exposes raw calls.
Key terms: ONNX Runtime (ORT), enable_profiling, end_profiling, JSON file, Ort_perf_view, treemaps, ONNX operator, execution provider, per-operator latency
10.2 Transforming raw ONNX profiling data into insights
The section shows how to turn raw ONNX profiling data into charts that reveal where an ONNX model spends inference time.
- Profiling Data Transformation: Trace how the example moves from a profiled ONNX inference run to data suitable for analysis: save the raw JSON profiling data, load it, and use OnnxWholeSession to make a cleaner tabular representation.
Why it matters: The tabular data is the starting point for the performance comparisons that follow. - Operator Performance Views: Describe the three views built by aggregating profiling data by ONNX operator type: total duration, invocation count, and percentage of total inference time. Explain how they help identify opportunities for graph optimization.
Why it matters: The three views make costly or frequently invoked operators easier to spot. - Cross-Model Profiling Reuse: Explain why the section demonstrates the analysis with a simple converted linear regression model and how the same data-cleaning, aggregation, and visualization approach is used for ONNX GPT-2 small.
Why it matters: The simple example teaches an analysis workflow the reader can carry over to an ONNX LLM.
Key terms: raw profiling data, JSON file, OnnxWholeSession, tabular format, ONNX operator type, total duration, count of occurrences, percentage of total inference time, graph optimization, ONNX-converted model
10.3 Optimization of ONNX graphs for LLMs
The section shows how to inspect and test ONNX graph changes, then uses offline optimization of GPT-2 to illustrate their effects on graph structure and performance.
- ONNX graph evaluation and inspection: Explain how ReferenceEvaluator helps inspect a full ONNX graph or an individual node, and describe what increasing its verbosity reveals.
Why it matters: Inspecting intermediate computations helps you locate a problem before changing the graph. - Custom operator fusion: Explain the section’s EyeLike-and-Add fusion example and how ReferenceEvaluator evaluates the resulting custom operator.
Why it matters: A fused graph must remain executable and testable even when it contains an operator ONNX does not provide by default. - Fusion quality and speed checks: Describe how the section checks whether its fused model preserves predictions and improves execution speed.
Why it matters: A faster graph is useful only if it still gives the intended results. - GPT-2 offline graph optimization: Describe how the section applies the ORT optimizer to an ONNX-converted GPT-2 model and what graph and profiling changes it reports.
Why it matters: Comparing optimized and original graphs connects specific graph changes to measured LLM inference performance.
Key terms: ReferenceEvaluator, individual nodes, verbose, intermediate outputs, fusion, EyeLike, Add, AddEyeLike, OpRun, _run
10.4 Summary
Where this is used later
- Benchmarking various models (chapter 11) · Earlier work on optimizing ONNX graphs supplies one of the model versions compared here.
Concepts explained in this chapter
- ReferenceEvaluatorThe ONNX Runtime reference evaluator class used to execute ONNX graphs or individual nodes in a Python environment.
- kernel fusionAn optimization that combines adjacent operators into one to reduce overhead and improve inference latency.
- profilingThe practice of collecting runtime measurements for model execution to identify bottlenecks and optimization opportunities.
- ORT for PythonThe Python interface to ONNX Runtime used here to evaluate ONNX models, nodes, and intermediate results.
- ONNX RuntimeA runtime library that efficiently executes ONNX models across hardware targets such as CPUs, GPUs, and edge devices.
- InferenceSessionAn ONNX Runtime class and execution context for loading models, running inference, and applying runtime optimizations.
- ONNXA serialized machine-learning model represented in the ONNX format for portable execution outside the training framework.
- execution providerAn ONNX Runtime backend that assigns graph nodes to a specific hardware/runtime implementation such as CPU, CUDA, or ROCm.
- int4 quantizationA quantization method that represents model weights or activations with 8-bit numbers to reduce model size and improve efficiency.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.