The replicated service pattern scales stateless model serving when one server cannot keep up with growing request traffic. Run equivalent model-server replicas, each able to retrieve the trained model and independently process an inference request; add replicas as demand grows. section 4.2 section 4.2.2
Figure 4.7 A diagram showing how a loader balancer is used to distribute the requests evenly across model server replicas
A load balancer sits before equivalent replicas and assigns incoming requests across them, so separate requests can be processed without interfering with one another. section 4.2.2
Operate the pattern as a request-routing and availability design:
- Place a load balancer in front of replicas. It can use round robin, least connection, or hashing to select a target. section 4.2.2
- Keep replica compute resources identical so the load balancer can distribute requests correctly and evenly. section 4.3
- Use readiness probes as well as liveness checks: a replica must establish its remote model-storage connection before receiving traffic. section 4.2.3
- Plan for the load balancer itself to become overloaded if its request-processing work is expensive. section 4.2.2
- Choose sharded services rather than replicas alone when one request exceeds an individual replica’s memory capacity. section 4.2.3 section 4.3
Autoscaling model inference Pods
A KServe InferenceService can autoscale with the Knative Serving autoscaler. With concurrency as the scaling metric and a target of one concurrent request per instance, additional inference Pods start as in-flight requests increase. Load-test traffic with hey and inspect Pods with kubectl. section 9.3.2 section 9.5
Replicas also improve availability: if one model server crashes or is preempted, remaining ready replicas can continue serving requests. This supports uptime objectives such as three-nines availability, but does not make oversized individual requests processable. section 4.2.3 section 4.3
