liveBook · Manning

ORT for Python

As explained in Domain-Specific Small Language Models

The Python interface to ONNX Runtime used here to evaluate ONNX models, nodes, and intermediate results.

ORT for Python evaluates ONNX models as full graphs, single nodes, or intermediate steps, which helps expose inefficiencies after conversion from other frameworks or model versions. section 10.3 section 10.3

It is used to:

  • run a complete ONNX graph for a model. section 10.3
  • inspect one node at a time, including custom nodes. section 10.3
  • trace step-by-step execution with increasing verbosity. section 10.3
  • test optimized nodes and compare outputs and timing. section 10.3

Linear regression graph evaluation

A linear regression graph is evaluated as Add(MatMul(X, A), B), then reworked with EyeLike and a custom AddEyeLike node to compare outputs and runtime. section 10.3

from onnx.reference import ReferenceEvaluator

sess = ReferenceEvaluator(onnx_model)

Creates a ReferenceEvaluator session from an ONNX model, the basic entry point for graph evaluation. section 10.3

sess = ReferenceEvaluator(onnx_model, verbose=verbose)

Enables verbose execution tracing on a model session for progressively more detailed intermediate visibility. section 10.3

How Domain-Specific Small Language Models covers it

How deep it goes
Covers full-graph runs, node-level evaluation, verbosity levels, and custom-node comparison, but stops short of broader optimization theory.
Examples
Uses linear regression, a custom AddEyeLike node, and GPT-2 graph optimization with before-and-after figures and timings.
What it assumes
Assumes ONNX graphs and operators are already familiar, while introducing ReferenceEvaluator and verbose tracing from zero.
What you can do after
Apply it via ReferenceEvaluator to inspect outputs, debug graph structure, and compare optimized and original models.
How it connects
Connects ONNX, ReferenceEvaluator, custom nodes, EyeLike, Add, MatMul, and ONNX Runtime transformers optimization.
Cover of Domain-Specific Small Language Models
liveBookmanning.com
Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.