liveBook · Manning

ORTModelForCausalLM

As explained in Domain-Specific Small Language Models

The Optimum ONNX Runtime class for loading a causal language model in ONNX Runtime format.

ORTModelForCausalLM loads a causal language model into ONNX Runtime format through Optimum, then can save and quantize it for CPU inference. section 7.2.1

It is chosen here because CodeGen is a causal LM, and the model is downloaded from the Hugging Face Hub with an execution provider set at load time. section 7.2.1

Typical workflow in this material:

  • Export a supported causal LM to ONNX with a provider such as CPUExecutionProvider. section 7.2.1
  • Run text generation through a Transformers pipeline and benchmark latency and throughput. section 7.2.1

CodeGen 350M mono on CPU

The example loads Salesforce/codegen-350M-mono as an ORTModelForCausalLM with export=True and provider="CPUExecutionProvider", then saves and quantizes it.

How Domain-Specific Small Language Models covers it

How deep it goes
Explains the load-export-save-quantize workflow and benchmark setup, but stops short of internal ONNX Runtime mechanics.
Examples
Uses Salesforce/codegen-350M-mono, CPUExecutionProvider, a text-generation pipeline, and latency-throughput benchmark figures.
What it assumes
Assumes a causal LM and basic Hugging Face model loading; it introduces Optimum-based ONNX export from there.
What you can do after
Apply it to export, save, quantize, and benchmark a causal LM in ONNX Runtime on CPU.
How it connects
Connects ONNX Runtime, Optimum, Hugging Face Hub, Transformers pipeline, ORTQuantizer, and AutoQuantizationConfig.
Cover of Domain-Specific Small Language Models
liveBookmanning.com
Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.