Chapter 17: Deploying to production
The introduction explains why deployment choices depend on how people need to access a model and where it needs to run.
After this chapter you will be able to
- Interactive model access options. Distinguish a quick interactive model demo from a network service that exposes a model through an API, using Gradio and FastAPI as the section’s examples.
- Deployment beyond Python services. Explain why a deployment might use ONNX, C++, or a mobile device instead of only a Python-based service.
- Function-based Gradio interfaces. Explain how Gradio creates a web interface from a function that takes inputs and returns outputs, including the text-input and text-output example.
- Text-to-image Gradio demo. Describe how the section adapts a pretrained Stable Diffusion model for a Gradio demo and how that demo can be shared.
- Text Generation API Contract. Describe how the /generate POST route accepts the beginning of a sentence and returns its predicted continuation as JSON. Explain that FastAPI parses the incoming JSON into a Pydantic model defining the expected input.
- Model Inference Setup. Explain how the server prepares a pretrained model for requests: it loads the architecture and weights, can move the model to an available GPU, sets evaluation mode, and runs inference without gradient calculations.
- Asynchronous request handling. Explain how an asynchronous FastAPI endpoint can handle other requests while one request waits for I/O, and identify async def as the change used to define such an endpoint.
- Efficient model inference. Explain why asynchronous request handling is not enough for efficient model inference: GPU requests are often better batched than run individually or in parallel, and the section calls for model execution on a second thread without being held back by Python’s GIL.
- Token streaming for responsiveness. Describe why the server should stream generated text token by token instead of waiting for the complete response.
- Batch formation and decoupling. Explain why request handling is separated from model execution and when the worker processes a batch.
- Per-request batched generation. Trace how the worker generates tokens for a batch and associates output with individual requests.
- Incremental response streaming. Explain how generated tokens reach clients before their full responses are complete.
- Compilation for faster inference. Explain how torch.compile can speed up an existing PyTorch model with little code change, and why the result varies by model and hardware.
- Selective precision and tradeoffs. Explain why integer quantization cannot be applied to every part of a model with a single int8 conversion, and describe the section's selective approach and its tradeoffs.
- Reasons for model export. Explain why a production model may need to be exported rather than run directly through PyTorch abstractions, including deployment outside Python or on different hardware.
- Exported model representation. Describe what exporting converts a model into and distinguish that representation from low-level hardware instructions.
- ONNX Model Interoperability. Explain how ONNX represents a model as operations and parameters that another runtime can execute, and why execution depends on support for the model’s operations.
- PyTorch Model Export to ONNX. Describe how to export a PyTorch model with a dummy input and what can be done with the resulting ONNX file.
- Model tracing and capture. Explain that torch.export takes an nn.Module and an example input to trace a computation graph, and describe TorchDynamo's role in capturing operations during Python execution.
- Compiled deployment workflow. Trace the path from PyTorch model code through export to compilation and execution, including how AOTInductor packages compiled code, how that package is loaded for inference, and the stated input-shape caveat.
- Shared model execution flow. Explain that the flow from user model code through an exported model to execution is logically similar to what happens in torch.compile.
- Full graph capture limits. Explain why torch.export can fail on a model whose conditional or loop depends on input data, and why tracing only the example input’s path would not represent the whole model.
- Graph breaks and fallback. Describe how torch.compile handles operations or control paths it cannot compile by splitting the graph and falling back to regular Python, while retaining the model’s dynamic behavior.
- Profiler performance measurements. Explain how torch.profiler helps identify bottlenecks during training or inference, and describe the execution-time, memory-usage, and operation-count metrics it captures.
- Execution trace interpretation. Describe what a profiler trace reveals beyond a summary table, including CPU and GPU timelines, and explain how custom labels identify model layers.
- PyTorch models beyond Python. Explain that PyTorch models can run outside Python by using other offerings from PyTorch’s Ecosystems, including in C++ applications and on mobile devices.
- LibTorch role and tradeoffs. Explain that LibTorch provides direct C++ access to PyTorch functionality through an API designed to resemble Python’s, and describe when the section recommends using it.
- C++ inference and training. Describe the two kinds of LibTorch model use shown in the examples: linear-model inference and defining and training a neural network in C++.
- ExecuTorch on constrained devices. Explain why ExecuTorch is used for mobile phones and embedded systems: they may lack the resources to support the full range of PyTorch operations, so ExecuTorch optimizes model execution for them.
- ExecuTorch export and loading workflow. Trace the workflow described here: capture a model’s computational graph with torch.export, convert the exported program to ExecuTorch’s Edge IR, then load the package for use in a mobile application.
- Deployment across different environments. Describe the range of places PyTorch models can be deployed, from web services to C++ applications and mobile devices, and explain that the supporting tools are improving even though some aspects are still maturing.
What it builds on
- Reversing diffusion (how to sample) (chapter 10) · The earlier discussion of sampling images with a diffusion model.
- A dtype for every occasion (chapter 3) · Familiarity with tensor numeric types helps distinguish floating-point and integer representations.
- Our network as an nn.Module (chapter 8) · Defining a network as an nn.Module.
- The PyTorch nn module (chapter 6) · Familiarity with Python’s nn.Module makes the C++ equivalent meaningful.
Section by section
Introduction
The introduction explains why deployment choices depend on how people need to access a model and where it needs to run.
- Interactive model access options: Distinguish a quick interactive model demo from a network service that exposes a model through an API, using Gradio and FastAPI as the section’s examples.
Why it matters: The right way to make a model available depends on whether you need a shareable demo or a network service. - Deployment beyond Python services: Explain why a deployment might use ONNX, C++, or a mobile device instead of only a Python-based service.
Why it matters: Deployment is not limited to a Python web service: the target environment and users’ privacy needs can change the choice.
Key terms: Gradio, FastAPI, network service, ONNX, C++, mobile devices
17.1 Serving PyTorch models
The section introduces a hands-on path from a simple network server for sharing a model to examining its limitations, with a warning not to use an untrusted network.
17.1.1 Our model served by Gradio
This section shows how Gradio turns a Python function into a shareable model demo, using text-to-image generation as an example, while noting its production limits.
- Function-based Gradio interfaces: Explain how Gradio creates a web interface from a function that takes inputs and returns outputs, including the text-input and text-output example.
Why it matters: This lets you demonstrate and test a model without building a web interface from scratch. - Text-to-image Gradio demo: Describe how the section adapts a pretrained Stable Diffusion model for a Gradio demo and how that demo can be shared.
Why it matters: The same function-based approach can turn a pretrained image generator into an interactive demo that others can try. - Gradio production limitations: Distinguish Gradio's usefulness for quick demos and testing from its suitability for production applications.
Why it matters: A convenient demo interface should not be mistaken for a production serving solution.
Key terms: Gradio, function signature, input and output types, diffusers, Stable Diffusion, text prompt, demo.launch(share=True), production, high-performance, scalable
17.1.2 Our model behind a FastAPI server
This section shows how to serve a text-generation model through a FastAPI endpoint and test it locally.
- Text Generation API Contract: Describe how the /generate POST route accepts the beginning of a sentence and returns its predicted continuation as JSON. Explain that FastAPI parses the incoming JSON into a Pydantic model defining the expected input.
Why it matters: A clear request and response contract lets a client use the model through an API. - Model Inference Setup: Explain how the server prepares a pretrained model for requests: it loads the architecture and weights, can move the model to an available GPU, sets evaluation mode, and runs inference without gradient calculations.
Why it matters: These steps make the saved model ready to handle requests without doing training work. - Local API Testing: Describe how a client can try the running API through FastAPI's interactive /docs page or by sending a POST request with a JSON payload using curl.
Why it matters: Testing from a client checks that the server accepts input and returns the model's output.
Key terms: FastAPI, POST route, /generate, Pydantic model, JSON response, pretrained weights, .to('cuda'), model.eval(), torch.no_grad(), inference
17.1.3 What we want from deployment
The section identifies the main requirements for serving a PyTorch model efficiently and safely: concurrent request handling, efficient inference, streaming output, and bounded inputs.
- Asynchronous request handling: Explain how an asynchronous FastAPI endpoint can handle other requests while one request waits for I/O, and identify async def as the change used to define such an endpoint.
Why it matters: This distinguishes handling concurrent requests from making model inference itself faster. - Efficient model inference: Explain why asynchronous request handling is not enough for efficient model inference: GPU requests are often better batched than run individually or in parallel, and the section calls for model execution on a second thread without being held back by Python’s GIL.
Why it matters: Serving several requests at once helps only if the model computation is also arranged efficiently. - Token streaming for responsiveness: Describe why the server should stream generated text token by token instead of waiting for the complete response.
Why it matters: A user can start reading a response before generation is complete. - Bounded input safety: Explain the section’s limited safety concern: serving should guard against overflows and resource exhaustion, and fixed-size text inputs make that concern simpler than more complex inputs such as images.
Why it matters: Input size is one practical constraint to consider when exposing a model through an API.
Key terms: FastAPI, async def, asynchronous endpoints, coroutines, model inference time, batch requests, GPU, second thread, global interpreter lock (GIL), streaming the response
17.1.4 Request batching and streaming responses
This section shows how a FastAPI server can batch incoming model requests while streaming generated tokens back to each client.
- Batch formation and decoupling: Explain why request handling is separated from model execution and when the worker processes a batch.
Why it matters: This explains how several requests can share model execution without making the API stop accepting new work. - Background worker coordination: Describe how the server starts and shuts down its worker and how shared request and result data are protected.
Why it matters: The server and worker share work and results, so their lifecycle and access to shared data need coordination. - Per-request batched generation: Trace how the worker generates tokens for a batch and associates output with individual requests.
Why it matters: Batching model execution still requires keeping each generated response tied to its request. - Incremental response streaming: Explain how generated tokens reach clients before their full responses are complete.
Why it matters: Clients can see output promptly instead of waiting for the complete generated response.
Key terms: request batching, queue, model worker, full batch, maximum time, lifespan event, model_worker, shutdown_event, inference_queue, results dictionary
17.1.5 How to make PyTorch models even faster
This section presents practical ways to speed up PyTorch inference or reduce model footprint, then points to existing serving tools.
- Compilation for faster inference: Explain how torch.compile can speed up an existing PyTorch model with little code change, and why the result varies by model and hardware.
Why it matters: Compilation is a potential inference speedup that does not require rewriting the model, but its benefit needs to be checked. - Model simplification versus quantization: Distinguish reducing the number of parameters or operations from reducing the footprint of each parameter and operation.
Why it matters: The two approaches reduce model cost in different ways, so they should not be treated as interchangeable. - Post-training quantization choices: Distinguish quantization-aware training from post-training quantization and explain why the section uses the latter for a pretrained model.
Why it matters: The choice depends on whether quantization can be incorporated into training or must be applied to a model that is already trained. - Selective precision and tradeoffs: Explain why integer quantization cannot be applied to every part of a model with a single int8 conversion, and describe the section's selective approach and its tradeoffs.
Why it matters: A substantial size reduction is possible, but applying lower precision safely requires attention to which parts of the model support it. - Existing serving tools: Describe when the section suggests using existing serving technologies and what vLLM and LitServe offer.
Why it matters: Existing tools may supply serving capabilities without requiring a custom implementation for every model.
Key terms: torch.compile, just-in-time (JIT) compilation, kernel fusion, operator reordering, distillation, quantization, 32-bit, 8-bit, quantization-aware training, post-training quantization
17.2 Exporting models
The section explains why production deployment may call for exporting a PyTorch model and what the export produces.
- Reasons for model export: Explain why a production model may need to be exported rather than run directly through PyTorch abstractions, including deployment outside Python or on different hardware.
Why it matters: These constraints explain why running a model directly in Python is not always the right deployment choice. - Exported model representation: Describe what exporting converts a model into and distinguish that representation from low-level hardware instructions.
Why it matters: It clarifies what an exported model provides—and what work may still remain before it runs on particular hardware.
Key terms: export, GIL, embedded systems, standardized representation, exporting a model
17.2.1 Interoperability beyond PyTorch with ONNX
This section explains how ONNX lets PyTorch models run outside PyTorch and shows how to export a model for an ONNX-compatible runtime.
- ONNX Model Interoperability: Explain how ONNX represents a model as operations and parameters that another runtime can execute, and why execution depends on support for the model’s operations.
Why it matters: The ONNX file is useful only when the destination can understand the computations your model requires. - PyTorch Model Export to ONNX: Describe how to export a PyTorch model with a dummy input and what can be done with the resulting ONNX file.
Why it matters: The dummy input supplies the shape and type needed to capture computations for deployment elsewhere.
Key terms: ONNX, ONNX-compatible runtime, operations, dummy input, torch.onnx.export, trace, ONNX file
17.2.2 PyTorch’s own export: torch.export
This section shows how torch.export captures a PyTorch model as a graph and how that graph can be compiled, packaged, and run for deployment.
- ONNX and torch.export roles: Distinguish why a developer would choose ONNX or torch.export: ONNX emphasizes compatibility with other frameworks and runtimes, while torch.export supports deployment optimization within the PyTorch ecosystem.
Why it matters: The deployment destination helps determine which export format to use. - Model tracing and capture: Explain that torch.export takes an nn.Module and an example input to trace a computation graph, and describe TorchDynamo's role in capturing operations during Python execution.
Why it matters: The captured graph is the representation that later export and compilation steps work with. - Exported graph representation: Describe what the section points out in the exported graph: parameters become arguments, layers appear as lower-level torch.ops.aten operations, and annotations show tensor dtype and shape.
Why it matters: Recognizing these changes makes an exported graph easier to inspect without mistaking it for the original model code. - Compiled deployment workflow: Trace the path from PyTorch model code through export to compilation and execution, including how AOTInductor packages compiled code, how that package is loaded for inference, and the stated input-shape caveat.
Why it matters: Export is an intermediate step: compilation and loading turn the captured model into something that runs in deployment.
Key terms: ONNX, torch.export, interoperability, torch.nn.Module, example input, traced graph, TorchDynamo, parameters, torch.ops.aten, dtype
17.3 Expanding on torch.compile
This introduction connects the previously discussed model export and execution flow to torch.compile before the next section examines their differences.
- Shared model execution flow: Explain that the flow from user model code through an exported model to execution is logically similar to what happens in torch.compile.
Why it matters: This connection gives you a starting point for comparing torch.compile with model export.
Key terms: torch.compile, exported model
17.3.1 Full graph capture vs. disjoint graphs
This section contrasts torch.export’s full-graph requirement with torch.compile’s ability to optimize models despite graph breaks, then explains the compilation process and its predecessors.
- Full graph capture limits: Explain why torch.export can fail on a model whose conditional or loop depends on input data, and why tracing only the example input’s path would not represent the whole model.
Why it matters: It explains why a model that runs normally may still fail to export. - Graph breaks and fallback: Describe how torch.compile handles operations or control paths it cannot compile by splitting the graph and falling back to regular Python, while retaining the model’s dynamic behavior.
Why it matters: This is the key reason torch.compile can optimize a model that torch.export cannot capture as one graph. - Earlier compilation tradeoffs: Distinguish eager execution, torch.jit.trace, and torch.jit.script in terms of their development or compilation tradeoffs, and state which tools the section recommends focusing on now.
Why it matters: The history clarifies why PyTorch sought compilation without giving up flexible, debuggable model code. - Compiled execution and backends: Trace the section’s high-level account of running a torch.compile model: reusing a cached compilation when possible, otherwise creating and compiling an FX graph, and executing generated code. Describe the default backend’s role.
Why it matters: It gives a practical picture of where compiled-model speedups come from without treating compilation as a black box.
Key terms: torch.export, full graph capture, static computation graph, dynamic control flow, torch.compile, TorchDynamo, Dynamo, graph breaks, disjoint graphs, regular Python
17.4 Understanding execution with torch.profiler
This section shows how torch.profiler reveals model performance and execution flow, including changes caused by compilation.
- Profiler performance measurements: Explain how torch.profiler helps identify bottlenecks during training or inference, and describe the execution-time, memory-usage, and operation-count metrics it captures.
Why it matters: Profiling gives you evidence about where a model spends its resources before you try to optimize it. - Execution trace interpretation: Describe what a profiler trace reveals beyond a summary table, including CPU and GPU timelines, and explain how custom labels identify model layers.
Why it matters: A trace lets you relate time spent on operations to the model layers and devices involved. - Compiled execution trace differences: Describe how the compiled model’s trace differs from the uncompiled trace, with forward operations grouped under a Compiled Region and fused GPU operations.
Why it matters: Comparing traces helps you see a concrete effect of compilation on execution.
Key terms: torch.profiler, execution time, memory usage, bottlenecks, trace, CPU execution timeline, GPU execution timeline, record_function(), Compiled Region, fused GPU operations
17.5 Using PyTorch outside of Python
Introduces PyTorch ecosystem offerings as a way to run models outside Python, including in C++ applications and on mobile devices.
- PyTorch models beyond Python: Explain that PyTorch models can run outside Python by using other offerings from PyTorch’s Ecosystems, including in C++ applications and on mobile devices.
Why it matters: This connects Python-based model development to deployment in other application environments.
Key terms: PyTorch’s Ecosystems, PyTorch models
17.5.1 LibTorch: PyTorch in C++
This section introduces LibTorch as a way to use PyTorch from C++, then outlines how to build and run examples and load a model compiled ahead of time.
- LibTorch role and tradeoffs: Explain that LibTorch provides direct C++ access to PyTorch functionality through an API designed to resemble Python’s, and describe when the section recommends using it.
Why it matters: It helps you decide whether using PyTorch from C++ is worth the tradeoff. - CMake build and execution: Describe how the section sets up, builds, and runs a LibTorch C++ application using CMake.
Why it matters: The build setup is the main extra hurdle before the C++ examples can run. - C++ inference and training: Describe the two kinds of LibTorch model use shown in the examples: linear-model inference and defining and training a neural network in C++.
Why it matters: These examples show that LibTorch supports both running models and writing training code in C++. - Ahead-of-time model loading: State what the final example achieves by compiling and saving a PyTorch model ahead of time.
Why it matters: It shows another route to using a PyTorch model in a C++ application.
Key terms: LibTorch, C++ API, Python API, CMake, CMAKE_PREFIX_PATH, build directory, inference, torch::nn::Module, Net, ahead-of-time (AOT) inductor
17.6 Going mobile: ExecuTorch
This section explains why ExecuTorch is useful on resource-constrained devices and outlines how an exported PyTorch model is prepared and run in a mobile application.
- ExecuTorch on constrained devices: Explain why ExecuTorch is used for mobile phones and embedded systems: they may lack the resources to support the full range of PyTorch operations, so ExecuTorch optimizes model execution for them.
Why it matters: It explains why deploying to a phone calls for a different approach from running PyTorch on a server. - ExecuTorch export and loading workflow: Trace the workflow described here: capture a model’s computational graph with torch.export, convert the exported program to ExecuTorch’s Edge IR, then load the package for use in a mobile application.
Why it matters: It connects the exported PyTorch model to the package a mobile application can use. - Alternative native mobile integration: Distinguish ExecuTorch’s simplified mobile deployment path from compiling LibTorch and C++ for Android or iOS and bridging to application code.
Why it matters: It clarifies the practical complexity that ExecuTorch is intended to avoid.
Key terms: ExecuTorch, mobile phones, embedded systems, PyTorch operations, torch.export, computational graph, Edge IR, runtime environment, LibTorch, JNI
17.7 Conclusion
The section concludes that PyTorch offers deployment paths for varied environments, supported by an evolving ecosystem.
- Deployment across different environments: Describe the range of places PyTorch models can be deployed, from web services to C++ applications and mobile devices, and explain that the supporting tools are improving even though some aspects are still maturing.
Why it matters: The deployment approach can be chosen to fit where a model needs to run.
Key terms: Gradio, FastAPI, ONNX, LibTorch, ExecuTorch, torch.compile
17.8 Exercises
17.9 Summary
Concepts explained in this chapter
- torch.compile()torch.compile() is a PyTorch API that compiles functions or modules into optimized executable code for execution.
- torch.exportA PyTorch API for exporting models for deployment within the PyTorch ecosystem, including ahead-of-time compilation.
- torch.fxA graph representation produced by TorchDynamo that stores recorded instructions for later optimization or export.
- PyTorch Dynamo compilerA PyTorch runtime graph-capture tool that intercepts Python execution to trace and optimize computation graphs.
- batch sizeThe number of training examples processed together in one update step.
- streamingSending generated output incrementally to a client as it becomes available rather than waiting for the full result to finish.
- torch.onnx.exportA PyTorch export API that writes a model into ONNX format, optionally using Dynamo-based capture.
- quantizationA model optimization technique that reduces numerical precision of parameters and operations to lower memory use and often speed up inference.
- full graph captureA compilation analysis mode that records an entire model computation as one graph instead of fragmenting it into pieces.
- ONNXA vendor-neutral standard and file format for representing neural network computations so models can be executed in different runtimes and devices.
- GradioA Python library for quickly building interactive web demos and prototypes around machine learning models.
- computation graphA graph representation captured from model execution that records operations and data flow.
- AOTInductorPyTorch's ahead-of-time compiler that turns exported graphs into optimized executable code.
- torch.nn.ModuleThe PyTorch base class for defining neural-network models as Python modules with parameters, submodules, and a forward method.
- model workerA background execution component that pulls requests from a queue and runs model inference on them.
- queueA first-in, first-out structure that stores incoming work items until a worker processes them.
- token generation loopAutoregressive generation that produces one token at a time and exposes each token as soon as it is created.
- ExecuTorchA PyTorch ecosystem library/runtime for deploying models efficiently on constrained devices such as mobile and embedded systems.
- deploymentThe process of putting a trained model into a real runtime environment where it serves users or applications.
- kernel fusionAn optimization that combines multiple operations into fewer executable units to reduce overhead and improve speed.
- just-in-time compilationA compilation strategy that performs code optimization or translation during execution rather than before runtime.
- torch::nn::ModuleThe C++ PyTorch base class for defining neural-network modules.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.