MLCEngine

As explained in Domain-Specific Small Language Models

MLC LLM’s unified inference engine object used to run compiled models and provide OpenAI-compatible chat completions.

MLCEngine is MLC LLM’s unified high-performance inference engine for running compiled language models across platforms. It exposes OpenAI-compatible chat completions through Python, JavaScript, iOS, Android, and a REST server, using one engine and compiler across deployment targets. section 11.3

A typical deployment path prepares the model before constructing the engine:

• Convert Hugging Face model weights to MLC format with a selected quantization type and target device. section 11.3
• Generate the chat configuration with the model’s conversation template. section 11.3
• Compile the configuration into a platform-specific model library. section 11.3
• Instantiate MLCEngine with the converted model directory and compiled model library, then call chat completions locally or through REST. section 11.3 section 11.5

Python chat completion

from mlc_llm import MLCEngine\nengine = MLCEngine(model="./dist/model", model_lib="./dist/libs/model-cuda.so")\nresponse = engine.chat.completions.create(messages=[{"role": "user", "content": "What is the meaning of life?"}], model="./dist/model", stream=False) section 11.3

Figure 11.3 The end-to-end workflow for MLC LLM, from model conversion to deployment on the supported platforms’ native runtimes

Follow the end-to-end path from model conversion through compilation and deployment on supported native runtimes. section 11.3

How Domain-Specific Small Language Models covers it

How deep it goes

Defines the engine and demonstrates Python and REST inference, but stops short of explaining runtime internals.

Examples

Uses quantized RedPajama, Python chat completion, streaming output, and a streamed REST request.

What it assumes

Assumes a Hugging Face model and prepared platform dependencies, then teaches conversion, compilation, and inference.

What you can do after

Operate/configure a compiled model through Python chat completions or an OpenAI-compatible REST endpoint.

How it connects

Connects model conversion, quantization, configuration generation, compilation, OpenAI-compatible APIs, and REST serving.

liveBookmanning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

• MLC LLMused by · An open-source ML compiler and deployment engine for serving large language models across multiple platforms.
• OpenAI-compatible APIequivalent to · An OpenAI-compatible API follows enough of the OpenAI API’s request and response conventions to support clients designed for that API.
• OpenAI APIequivalent to · The OpenAI API is a programming interface that allows software applications to send requests to OpenAI models and services and receive responses.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.