Scalability keeps an ML system usable as request volume and resource demand grow. A model that works in evaluation can fail under heavy production load, making performance characterization and robust testing essential. section 2.2 section 10.1
Treat scalability as part of an operational deployment pattern:
- Identify whether inference is batch, real-time, or both before selecting a scaling approach. section 10.1
- Run load tests against expected demand, then prepare auto-scaling for spiky workloads. section 1.1.2
- Monitor requests per second, HTTP status codes, latency, resource use, drift, and business impact after deployment. section 1.1.2 section 10.1
- Combine scaling with versioning, rollback, observability, and automated deployment so capacity changes do not compromise reliability. section 1.1.2 section 10.9
Production inference service
A model exposed through a REST API can be containerized with Docker and deployed on AWS or GCP. Load testing reveals whether it handles expected demand; auto-scaling addresses spikes, while monitoring tracks RPS and HTTP status codes. section 1.1.2
Figure 1.2 The development/staging/production phase of the ML life cycle
Use the ML life-cycle diagram to connect development and staging decisions with production deployment, monitoring, and retraining. section 1.1.2
