A deployment service turns a trained model into a production capability. It configures the resources required to run inference, connects model-serving components, and exposes results through APIs or data pipelines. section 3.4.8
It can provide boilerplate for REST or gRPC endpoints, batch jobs, and streaming pipelines. Standard serving tools also connect model health checks, metadata, prediction, and explanation methods with CI/CD workflows, testing, rollbacks, and deployment monitoring. section 3.4.8
Choose the serving path according to the required balance of standardization and customization:
- Use NVIDIA Triton Inference Service, MLServer, Seldon, or BentoML when standardized deployment and generated service structure reduce implementation work. section 3.4.8
- Use FastAPI when application-specific flexibility justifies leaving the more automated serving path, while retaining as much boilerplate and automation as possible. section 3.4.8
- Treat the service as the bridge between MLOps systems and DevOps infrastructure rather than merely as an HTTP interface. section 3.4 section 3.4.8
Figure 3.7 A high level view of an MLOps infrastructure with LLMs in mind. This diagram attempts to cover the full picture and the complexity of the many tools involved to make ML models work in production.
Use the infrastructure overview to see how deployment services connect model registries, vector databases, pipelines, monitoring, and broader DevOps systems. section 3.4
