liveBook · Manning

CPUExecutionProvider

As explained in Domain-Specific Small Language Models

The ONNX Runtime execution provider that runs model computation on CPU-backed execution.

CPUExecutionProvider selects CPU-backed execution for an ONNX Runtime model when downloading or running it through Optimum. section 7.2.1

It appears in a CPU benchmarking workflow that compares PyTorch, ONNX conversion, and quantized ONNX versions on the same prompt and metrics. section 7.2.1

Typical use here centers on exporting and benchmarking a causal language model on CPU:

  • Pass provider="CPUExecutionProvider" when creating an ORT model from pretrained weights. section 7.2.1
  • Save the converted model, quantize it dynamically, and compare latency and throughput against the vanilla model. section 7.2.1
  • Swap the pipeline model and provider tags to reuse the same inference benchmark code. section 7.2.1

CPU provider in model export

ORTModelForCausalLM.from_pretrained(model_id, export=True, provider="CPUExecutionProvider") loads the model for ONNX export on CPU before saving and quantization. section 7.2.1

How Domain-Specific Small Language Models covers it

How deep it goes
Defines the provider only by its deployment role and shows usage in export and benchmark code, not runtime internals.
Examples
Uses CodeGen 350M mono, ORTModelForCausalLM.from_pretrained(..., provider="CPUExecutionProvider"), and CPU benchmark runs.
What it assumes
Assumes familiarity with Hugging Face model loading and ONNX export, and teaches provider selection from the example.
What you can do after
Apply it via Optimum to export, save, quantize, and benchmark a model on CPU.
How it connects
Connects to ONNX Runtime, Optimum, ORTModelForCausalLM, ORTQuantizer, and Transformers pipeline.
Cover of Domain-Specific Small Language Models
liveBookmanning.com
Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.