How deeply does LLMs in Production cover ONNX Runtime?

Defines ONNX Runtime’s role and deployment rationale but stops short of explaining its internal optimization algorithms.

ONNX Runtime

As explained in LLMs in Production

ONNX Runtime is a cross-platform inference engine that optimizes and executes models represented in ONNX format.

When TensorRT does not support a model or hardware setup cleanly, ONNX Runtime offers a practical alternative for optimizing and running ONNX models. ONNX is the standardized interchange format; ONNX Runtime is the inference engine. section 6.1.1

Use ONNX Runtime when deployment requires:

Porting models trained with TensorFlow, PyTorch, Keras, or MXNet into a standardized representation. section 6.1.1
Running inference across CPUs, GPUs, and specialized accelerators rather than targeting one hardware vendor. section 6.1.1
Avoiding extensive reengineering when moving models between tools, platforms, and environments. section 6.1.1

Export and optimize with Optimum

Hugging Face Optimum can export a model to ONNX and apply optimization: `optimum-cli export onnx --model WizardLM/WizardCoder-1B-V1.0 ./models_onnx --optimize O1`. The resulting ONNX model can then be optimized and executed with ONNX Runtime. section 6.1.1

How LLMs in Production covers it

liveBook manning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Domain-Specific Small Language ModelsA runtime library that efficiently executes ONNX models across hardware targets such as CPUs, GPUs, and edge devices.
liveBook manning.com

Machine Learning System DesignONNX Runtime is a cross-platform runtime for executing ONNX machine-learning models with hardware-specific optimizations.

Compare how 3 books explain ONNX Runtime →

Related concepts

ONNXuses · ONNX is an open standard for representing machine-learning models in a portable format that supports exchange and execution across frameworks and runtimes.
execution providerused by · An execution provider is an ONNX Runtime backend that assigns and executes model-graph operations on a specific hardware or software target.
TensorRTuses · TensorRT is NVIDIA’s software development kit and inference runtime for optimizing and executing machine-learning models on NVIDIA hardware.
profilinguses · Profiling is the measurement and analysis of a program’s runtime behavior and resource use to characterize performance and identify bottlenecks.
generationuses · Generation is the computational production of outputs from supplied inputs or context, often by applying a model to produce predictions or new content.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.