liveBook · Manning

ReferenceEvaluator

As explained in Domain-Specific Small Language Models

The ONNX Runtime reference evaluator class used to execute ONNX graphs or individual nodes in a Python environment.

ReferenceEvaluator is an ONNX reference evaluator for executing a complete ONNX graph or an individual node from Python. It supports inspecting intermediate results, making it useful when conversion or optimization introduces inefficiencies. section 10.3

Use it to:

  • Run a model with input feeds and inspect its outputs. section 10.3
  • Evaluate one node independently to isolate behavior or investigate graph inefficiencies. section 10.3
  • Increase verbosity from level 1 to 4 to expose operations, values, intermediate outputs, and intermediate inputs. section 10.3 section 10.4
  • Register Python implementations of custom operators by extending OpRun and passing them through new_ops. section 10.3

Checking an optimized graph

Run the original and optimized models with identical feeds, compare their outputs, and measure execution time. In the demonstrated linear-regression graph, replacing EyeLike plus Add with AddEyeLike produced identical outputs and reduced the reported time for 10,000 runs from 1.1428 to 0.8636. section 10.3

from onnx.reference import ReferenceEvaluator

sess = ReferenceEvaluator(onnx_model)

Use this minimal construction to create an evaluator from an ONNX model before supplying feeds to run. section 10.3

How Domain-Specific Small Language Models covers it

How deep it goes
Explains full-graph, single-node, verbose, and custom-operator evaluation, but defers broader runtime profiling to other ONNX Runtime tools.
Examples
Demonstrates linear regression, MatMul inspection, verbose traces, AddEyeLike fusion, output comparison, and timing.
What it assumes
Assumes an ONNX model or node and prepared Python input arrays; teaches evaluation detail through feeds, verbosity, and registered operators.
What you can do after
Diagnose/debug with it by running ONNX graphs or nodes, tracing intermediate values, testing custom operators, and comparing optimized outputs.
How it connects
Connects ONNX graphs with node evaluation, custom operators, operator fusion, and performance profiling.
Cover of Domain-Specific Small Language Models
liveBookmanning.com
Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.