Chapter 11: Deployment and serving
The chapter introduces a selective survey of popular tools and environments for deploying and serving small, customized language models.
After this chapter you will be able to
- PagedAttention cache management. Explain how PagedAttention changes KV-cache storage and why that helps LLM serving.
- vLLM model initialization. Explain what creating a vLLM LLM object does in the text-generation example and why the model is then ready for inference.
- Text completion and sampling. Describe how the example starts text completion and where it configures the sampling strategy.
- Chat inference options. Distinguish the chat example from text completion, including its method, supported prompt grouping, and optional template.
- Online vLLM server setup. Describe how to start a vLLM HTTP server with the serve CLI command, including the model ID, precision format, and API authorization token.
- Online serving benchmarks. Describe why to benchmark a deployed vLLM model, the preparation and inputs for the example benchmark, and the kinds of results it reports.
- Routes and server access. Describe how a FastAPI application defines GET and POST routes, starts a server, and makes its API and documentation available.
- LLM inference endpoint structure. Trace the FastAPI text-generation example from request and response classes through model initialization to a POST route that returns generated text.
- Inference benchmarking priority. Explain why model inference is the focus when comparing models for FastAPI serving, and distinguish that comparison from benchmarking the web application.
- Comparable model latency tests. Describe the model versions and measurement approach used to compare inference latency.
- Latency results and candidate. State how latency differed by prompt length and justify the deployment candidate chosen from the results.
- ONNX text generation function. Describe how the custom generate function turns a prompt into text using the ONNX-optimized GPT-2 small model.
- FastAPI request and route. Describe how the web app accepts generation inputs and exposes the selected model through FastAPI.
- Cross-platform MLC LLM engine. Explain what MLC LLM and MLCEngine provide for deploying and serving language models across platforms.
- Model preparation and compilation. Trace the section's sequence from a Hugging Face model to a model ready for inference on a target device, distinguishing preparation from compilation.
- TensorFlow-based Android deployment requirements. Explain why Android deployment examples using TensorFlow Lite or MediaPipe may require extra work for a PyTorch SLM.
- Android deployment workflow. Trace the summarized MLC LLM workflow from preparing the local build environment through installing the app and model weights on an Android phone.
- mllm build and chat workflow. Describe how to build mllm from source for Linux, macOS, or Android, then download a converted model and start a chat session with its model-specific script.
- Android Python environment setup. Describe how to prepare an Android device for Python-based SLM inference: enable developer options, install Termux and Python, create and activate a virtual environment, and install PyTorch and Transformers.
- On-device Transformers inference. Explain how the prepared Python environment is used to run a Transformers inference script on the device, using the section’s BERT classification example.
What it builds on
- Batching (chapter 4) · Uses the earlier topic of batching to frame vLLM’s emphasis on serving throughput.
- 4-bit quantization with ggml (chapter 6) · The earlier discussion of 4-bit quantization with ggml provides context for the GGUF format used here.
- Text completion (chapter 4) · Text generation from an input prompt.
- Optimization of ONNX graphs for LLMs (chapter 10) · Earlier work on optimizing ONNX graphs supplies one of the model versions compared here.
- The ONNX runtime (chapter 5) · The ONNX runtime used to create an inference session.
- 4-bit quantization (chapter 6) · Earlier treatment of quantization as an inference optimization technique.
Section by section
Introduction
The chapter introduces a selective survey of popular tools and environments for deploying and serving small, customized language models.
11.1 vLLM
Introduces vLLM as an LLM-serving library, explains the memory-management problems behind PagedAttention, and surveys its serving capabilities.
- PagedAttention cache management: Explain how PagedAttention changes KV-cache storage and why that helps LLM serving.
Why it matters: The way vLLM stores the KV cache is central to its serving efficiency. - Earlier serving memory problems: Describe the memory-management problems the section identifies in earlier LLM-serving systems.
Why it matters: These wasted or inflexible KV-cache allocations explain the problem vLLM was designed to address. - vLLM serving capabilities: Describe vLLM’s role and identify representative capabilities listed for the project without attributing unexplained mechanics to them.
Why it matters: The feature overview places vLLM’s memory optimization within a broader serving tool.
Key terms: PagedAttention, KV cache, memory fragmentation, throughput, batch size, contiguous memory allocation, internal fragmentation, external fragmentation, preallocation, memory sharing
11.1.1 Offline serving
This subsection demonstrates offline inference with vLLM through text completion, chat, and a quantized GGUF model.
- vLLM model initialization: Explain what creating a vLLM LLM object does in the text-generation example and why the model is then ready for inference.
Why it matters: The compact API hides several setup steps that must happen before offline inference. - Text completion and sampling: Describe how the example starts text completion and where it configures the sampling strategy.
Why it matters: It separates the call that generates a completion from the options that shape its sampling. - Chat inference options: Distinguish the chat example from text completion, including its method, supported prompt grouping, and optional template.
Why it matters: The same basic offline setup can serve a conversation use case with a different inference call. - Quantized GGUF chat workflow: Trace how the example uses an already quantized GGUF model for offline chat inference.
Why it matters: This example shows that a prequantized GGUF model fits into the same basic vLLM chat workflow.
Key terms: LLM class, generate, KV cache, generate method, SamplingParams, temperature, top_p, top_k, chat method, conversation prompt
11.1.2 Online serving
This section shows how to serve an SLM through vLLM’s OpenAI-compatible HTTP API and benchmark its online performance.
- Online vLLM server setup: Describe how to start a vLLM HTTP server with the serve CLI command, including the model ID, precision format, and API authorization token.
Why it matters: These are the basic pieces needed to put a model behind a vLLM HTTP endpoint. - OpenAI-compatible client calls: Explain how a client can call a running vLLM server and identify the OpenAI API capabilities the section names.
Why it matters: OpenAI API compatibility lets clients use a familiar interface to request outputs from the served model. - Online serving benchmarks: Describe why to benchmark a deployed vLLM model, the preparation and inputs for the example benchmark, and the kinds of results it reports.
Why it matters: Benchmarking the HTTP endpoint checks the performance clients will actually encounter before they receive access.
Key terms: vLLM, serve CLI command, Hugging Face Hub, OpenAI API, OpenAI Python client API, completions, chat completions, embeddings, benchmark, ShareGPT
11.2 FastAPI
This section introduces FastAPI’s API-building features and shows how a text-generation model can be served through a POST route and packaged in Docker.
- FastAPI framework features: Explain how FastAPI supports building ML APIs with Python type hints, data validation, nonblocking operations, and automatically generated interactive documentation.
Why it matters: These features make FastAPI a practical framework for exposing a model through an API. - Routes and server access: Describe how a FastAPI application defines GET and POST routes, starts a server, and makes its API and documentation available.
Why it matters: Routes and server access are the basic structure reused when the section adds model inference. - LLM inference endpoint structure: Trace the FastAPI text-generation example from request and response classes through model initialization to a POST route that returns generated text.
Why it matters: This is the point where a general web API becomes an endpoint for model inference. - Docker deployment packaging: Describe how the section’s Dockerfile packages the FastAPI text-generation app and starts its web server.
Why it matters: Packaging the app puts the server and its dependencies together for deployment.
Key terms: FastAPI, Starlette, Pydantic, Python type hints, Swagger UI, ReDoc, route, HTTP GET, HTTP POST, dev
11.2.1 Benchmarking various models
This subsection benchmarks four versions of a language model to choose a FastAPI deployment candidate based on inference latency.
- Inference benchmarking priority: Explain why model inference is the focus when comparing models for FastAPI serving, and distinguish that comparison from benchmarking the web application.
Why it matters: It keeps the model comparison focused on the main performance bottleneck without treating it as a complete application benchmark. - Comparable model latency tests: Describe the model versions and measurement approach used to compare inference latency.
Why it matters: Comparing versions across prompt lengths gives a more useful basis for choosing a deployment candidate than a single timing. - Latency results and candidate: State how latency differed by prompt length and justify the deployment candidate chosen from the results.
Why it matters: The measurements show why a smaller precision format should not be assumed to be the fastest serving choice.
Key terms: FastAPI, model inference, benchmarking, FP32, ONNX, FP16, warmup, average latency, prompt lengths, ONNX optimized model
11.2.2 Deploying the best-performing model with FastAPI
The section shows how to serve a selected ONNX-optimized GPT-2 small model through a FastAPI route.
- ONNX text generation function: Describe how the custom generate function turns a prompt into text using the ONNX-optimized GPT-2 small model.
Why it matters: This function supplies the generated text that the API route serves. - Asynchronous generation rationale: Explain why the generate function is declared async when the model is exposed through a web API.
Why it matters: It explains the section’s choice to make web-facing generation asynchronous. - FastAPI request and route: Describe how the web app accepts generation inputs and exposes the selected model through FastAPI.
Why it matters: The request class and route connect a caller’s inputs to the deployed text-generation capability.
Key terms: generate function, ONNX model, ORT inference session, async, LLM latency, FastAPI, GenerateRequest, Pydantic’s BaseModel, route
11.3 MLC LLM
This section introduces MLC LLM and walks through preparing an SLM for a target device, then using its Python API or REST server for inference.
- Cross-platform MLC LLM engine: Explain what MLC LLM and MLCEngine provide for deploying and serving language models across platforms.
Why it matters: The shared engine explains why the section's preparation workflow can lead to several serving options. - Model preparation and compilation: Trace the section's sequence from a Hugging Face model to a model ready for inference on a target device, distinguishing preparation from compilation.
Why it matters: Conversion prepares the model files, while compilation produces the library needed for the intended device. - Python inference engine choices: Distinguish how the section uses MLCEngine and AsyncMLCEngine for chat completions.
Why it matters: The engine choice determines whether a Python application uses synchronous chat or concurrent asynchronous generation. - REST server access and controls: Describe how the section serves a ported model through MLC LLM's REST API and how a client can discover or use the running server.
Why it matters: REST serving lets applications use the ported model without calling the Python inference engine directly.
Key terms: MLC LLM, MLCEngine, OpenAI-compatible API, Hugging Face Hub, mlc_llm_convert_weights, mlc_llm_gen_config, mlc-chat-config.json, mlc_llm_compile, AsyncMLCEngine, streaming responses
11.4 Deployment and inference on Android devices
The section introduces private, offline SLM deployment on Android and contrasts common TensorFlow-based examples with the PyTorch alternatives that follow.
- TensorFlow-based Android deployment requirements: Explain why Android deployment examples using TensorFlow Lite or MediaPipe may require extra work for a PyTorch SLM.
Why it matters: This explains why the section turns to PyTorch alternatives for private, offline Android deployment.
Key terms: TensorFlow Lite, MediaPipe, PyTorch, ONNX
11.4.1 MLC LLM framework
This section summarizes how to package and deploy an SLM to Android with MLC LLM, and notes the device-specific checks and development dependencies involved.
- Android deployment workflow: Trace the summarized MLC LLM workflow from preparing the local build environment through installing the app and model weights on an Android phone.
Why it matters: The sequence shows what must be prepared before an MLC LLM model can run in the Android app. - Device and tooling considerations: Explain why an Android deployment needs attention to the target device and why this MLC LLM approach adds development-environment work.
Why it matters: A reusable workflow does not remove the need to check performance on the actual phone or prepare the required build tools.
Key terms: MLC LLM, runtime, model libraries, template Android app, model weights, APK, target device, hardware specs, benchmark performance, optimize the model
11.4.2 MLLM framework
This subsection explains how to build mllm, run converted models on desktop or Android, and prepare custom models for resource-constrained devices.
- mllm build and chat workflow: Describe how to build mllm from source for Linux, macOS, or Android, then download a converted model and start a chat session with its model-specific script.
Why it matters: This is the path from obtaining mllm to running a model on a supported device. - Custom model format conversion: Explain what to do when a desired or custom model is not available in mllm format, including which checkpoint formats the supplied converter accepts.
Why it matters: Conversion lets a reader use a custom model instead of relying only on models already available in mllm format. - Resource-conscious model quantization: Explain why mllm offers quantization for constrained devices and state the INT4 modes available for mllm models, including converted custom models.
Why it matters: Quantization is an option for fitting mllm models to devices with limited resources.
Key terms: mllm, build.sh, Android NDK, Hugging Face Hub, bin directory, convert.py, PyTorch, safetensors, mllm format, INT4 quantization
11.4.3 Hugging Face Transformers
This section shows how to run a Hugging Face Transformers model on an Android device using Python in Termux, then choose and optimize a model for the device’s resources.
- Android Python environment setup: Describe how to prepare an Android device for Python-based SLM inference: enable developer options, install Termux and Python, create and activate a virtual environment, and install PyTorch and Transformers.
Why it matters: This setup lets you run a Python inference script directly on an Android device. - On-device Transformers inference: Explain how the prepared Python environment is used to run a Transformers inference script on the device, using the section’s BERT classification example.
Why it matters: The key deployment step is running the inference script on the Android device itself. - Device-aware model selection: Explain why an Android deployment needs a model that fits the device’s memory, processing power, and storage, and how benchmarking can guide later optimization.
Why it matters: A script that runs on a phone still needs a model and performance profile the phone can support.
Key terms: developer options, Termux, virtual environment, PyTorch, Transformers, on-device inference, bert-base-uncased, classification, Python inference script, on-device performance
11.5 Summary
Where this is used later
- Building a reasoning domain-specific SLM (chapter 15) · vLLM as an inference engine
- Combining a domain-specific SLM with RAG (chapter 13) · The Phi-3 Mini GGUF model previously used for offline serving.
Concepts explained in this chapter
- vLLMA high-throughput model serving library for running and benchmarking language models with efficient attention and API compatibility.
- MLC LLMAn open-source ML compiler and deployment engine for serving large language models across multiple platforms.
- MLCEngineMLC LLM’s unified inference engine object used to run compiled models and provide OpenAI-compatible chat completions.
- mlc_llm_compileMLC LLM’s command-line utility for compiling a model library according to a generated chat configuration and target device.
- mlc_llm_convert_weightsAn MLC LLM command that converts model weights into MLC format using specified quantization settings.
- mlc_llm_gen_configMLC LLM’s command-line utility for generating a chat configuration for a selected model.
- FastAPIA Python web framework for building APIs with automatic documentation, asynchronous request handling, and data validation support.
- PagedAttentionAn attention algorithm that stores KV-cache blocks noncontiguously to reduce fragmentation and improve memory flexibility during LLM serving.
- GPT-2A decoder-only Transformer language model used for text generation, here serving as the architecture underlying ProtGPT2.
- language modelA very large neural language model trained on broad text corpora to predict or generate language, used here for code generation and assistance.
- transformersThe Hugging Face library for working with pretrained transformer models, tokenizers, pipelines, and related training and inference utilities.
- quantizationMethods that reduce numeric precision or representational size of model values to improve efficiency, especially for inference; here the absmax method is a specific quantization procedure.
- InferenceSessionAn ONNX Runtime class and execution context for loading models, running inference, and applying runtime optimizations.
- attentionA sequence-modeling technique that weights relevant parts of an input more heavily than less relevant parts.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.