Optimum

As explained in Domain-Specific Small Language Models

A Hugging Face extension library for optimizing transformer models for training and inference on target hardware, including ONNX workflows.

How Domain-Specific Small Language Models covers it

How deep it goes

Explains high-level export and quantization workflows, but defers custom ONNX configurations for unsupported architectures.

Examples

Uses BERT and DistilBERT sequence classifiers plus CodeGen causal-language-model conversion, quantization, pipelines, and CPU benchmarks.

What it assumes

Assumes a pretrained Transformers model and tokenizer, while teaching the high-level path from model loading through ONNX export and quantization.

What you can do after

Operate/configure ONNX Runtime exports and quantization, then evaluate accuracy, latency, and throughput against an unoptimized model.

How it connects

Connects Transformers, ONNX, ONNX Runtime, quantization, model export, and inference benchmarking.

liveBookmanning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

• ONNXproduces · ONNX is an open standard for representing machine-learning models in a portable format that supports exchange and execution across frameworks and runtimes.
• TFLiteproduces · TensorFlow Lite, now known as LiteRT, is a lightweight framework and runtime for deploying and executing machine-learning models on mobile, embedded, and edge devices.
• CodeGencontrasts with · A pretrained code-generation transformer model family used here as the chapter’s running example for Python code generation and inference optimization.
• ONNX Runtimeuses · ONNX Runtime is an open-source, cross-platform inference engine for executing and optimizing machine-learning models represented in the ONNX format.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.