Horizontal scaling increases capacity by adding equivalent service instances instead of enlarging one machine. In model serving, replicated stateless servers handle growing request volume without requests interfering with one another. section 4.2.2
A practical model-serving arrangement follows this flow:
- Keep the model server stateless so each request can be processed independently. section 4.2.2
- Add exact server replicas with separate addresses as demand increases. section 4.2.2
- Distribute requests across replicas, allowing them to process model-serving work asynchronously. section 4.2.2
- Contrast this approach with vertical scaling, which adds computational resources to an existing machine. section 4.2.2
Video inference service
A service receives videos uploaded by users and sends inference requests to several identical model-server replicas. Each replica retrieves the trained model and processes a different request, while a load balancer distributes traffic across the fleet. section 4.2.2
