MLC LLM

As explained in Domain-Specific Small Language Models

An open-source ML compiler and deployment engine for serving large language models across multiple platforms.

MLC LLM is an open-source ML compiler and high-performance deployment engine for serving language models across platforms, including mobile and edge devices. Its MLCEngine provides one runtime behind OpenAI-compatible interfaces for REST, Python, JavaScript, iOS, and Android applications. section 11.3

An MLC LLM deployment follows a model-to-runtime pipeline:

• Obtain model weights and configuration files from the Hugging Face Hub, then convert the weights into MLC format with a chosen quantization setting and target device. section 11.3
• Generate the chat configuration, compile a platform-specific model library, and load both the converted model and library into MLCEngine. section 11.3
• Run completions through the OpenAI-style API or expose the model through an MLC LLM REST server. section 11.3
• Use batching, prompt caching, concurrency, and platform-specific compilation to address serving performance and deployment constraints. section 11.3

Converting and serving RedPajama

A Linux deployment can convert RedPajama INCITE Instruct 3B with q4f16_1 quantization for CUDA, generate its redpajama_chat configuration, compile a CUDA model library, and load it with MLCEngine. The same converted model can serve streamed chat completions at an OpenAI-compatible REST endpoint. section 11.3 section 11.5

Figure 11.3 The end-to-end workflow for MLC LLM, from model conversion to deployment on the supported platforms’ native runtimes

The end-to-end workflow diagram connects model conversion, compilation, the unified engine, and deployment on native platform runtimes. section 11.3

How Domain-Specific Small Language Models covers it

How deep it goes

Traces conversion, quantization, configuration, compilation, inference, and REST serving, while deferring Android implementation details to documentation.

Examples

Uses RedPajama INCITE Instruct 3B on Linux with CUDA and describes Android deployment on a Samsung S23 with Snapdragon 8 Gen 2.

What it assumes

Assumes a Hugging Face model and platform toolchains; Android builds additionally require Android Studio, Rust, Java, and the NDK.

What you can do after

Builds a converted and compiled model deployment, then operates it through Python, REST, or platform-specific application interfaces.

How it connects

Connects model conversion, quantization, Hugging Face Hub, MLCEngine, OpenAI-compatible APIs, REST serving, batching, and prompt caching.

liveBookmanning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

• MLCEngineuses · MLC LLM’s unified inference engine object used to run compiled models and provide chat completions.
• Dockerused by · Docker is a platform and ecosystem for building, distributing, and running applications in containers.
• vLLMalternative to · vLLM is an open-source library and serving engine for high-throughput, memory-efficient inference of large language models.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.