A training pipeline may work comfortably on one machine today and struggle as its dataset grows. An inference service may handle ordinary traffic but slow down during a holiday sale. Scalability is the ability to handle that growth or a sudden peak while maintaining acceptable operation. Plan for expected demand, not just current demand. section 10.3 section 15.1
For training, dataset size matters relative to the data type and available memory: a million tabular records and a million video clips pose very different problems. A useful warning sign is a dataset that no longer fits in one machine’s RAM. Two standard responses are: section 10.3
Choose a scaling approach according to the limit you have reached:
- Scale vertically by upgrading the training machine. This is simple, particularly with cloud resources, but eventually reaches the limits of one machine. section 10.3
- Scale horizontally by distributing work across machines. Multiple GPUs in one machine are a useful first step, but distributing work across machines is the larger change. section 10.3
Figure 10.4 Vertical scaling vs. horizontal scaling
Compare adding capacity to one machine with spreading work across machines. section 10.3
For inference, ask how many predictions the system must produce at its busiest point—and how quickly. Throughput measures predictions per unit of time; latency measures how promptly a prediction arrives. Retail traffic may spike predictably around major sales, while a viral app can create an unexpected burst. Peak capacity must preserve the latency and throughput the application needs, not merely keep servers running. section 15.1
Batch demand forecasting
A scheduled forecasting job reads inputs from S3 and writes predictions back to S3. AWS Batch and S3 provide horizontal scaling; auto-scaling groups adjust batch servers to demand. Because throughput matters more than immediate responses, the work can be parallelized without specialized hardware initially. Track job failures, duration, rows processed, resource use and prediction quality before deciding whether further optimization is needed. section 15.5.1
Real-time serving calls for a different check: enable autoscaling, monitor requests, resource usage and active instances, and configure alerts. A cache can reduce repeated inference for popular queries. Serverless inference can ease scaling for sporadic demand, but concurrency limits and model cold starts may make it unsuitable for strict latency targets. Start with the simplest design that meets measured requirements; cost, reliability and flexibility also compete with performance. section 15.5.2 section 15.3.2 section 15.6
