Model serving deploys a trained machine-learning model as an operational service that accepts new data and returns predictions or generated outputs. In enterprise GenAI systems, it supports responsive, scalable applications alongside LLMOps, observability, model management, reliability, performance, security, and compliance. section 11.4 section 11.4.7
Production serving requires coordinated infrastructure and operational controls:
- Scale resources with demand to preserve availability and cost-efficiency. section 11.4.7
- Use caching to reduce latency and load balancing to distribute requests evenly. section 11.4.7
- Plan for substantial skills, computing, operational, and specialized-hardware requirements, including GPUs with InfiniBand connectivity. section 11.4.7
- Manage inference metrics, latency, quotas, rate limits, observability, and model updates as part of deployment. section 11.4 section 11.4.7
Production model transition
An enterprise assistant can route incoming requests through balanced serving capacity, reuse cached results where appropriate, monitor inference latency and behavior, and move to a new model through a blue-green deployment to minimize downtime. section 11.4 section 11.4.7
Listing 11.3 docker-cmpose file for MLflow
services:
mlflow:
image: ghcr.io/mlflow/mlflow:latest
command: mlflow server --backend-store-uri /mlflow/mlruns
--default-artifact-root /mlflow/artifacts --host 0.0.0.0
ports:
- "5000:5000"
volumes:
- ./mlflow/mlruns:/mlflow/mlruns
- ./mlflow/artifacts:/mlflow/artifacts
prometheus:
image: prom/prometheus:latest
command: --config.file=/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- ./prometheus/data:/prometheus/data
depends_on:
- mlflow
A Docker Compose setup pairs MLflow with Prometheus, illustrating a concrete foundation for managing and monitoring deployed ML services. section 11.4.7
