The sharded service pattern handles large model-serving requests by dividing one workload into sub-requests, processing them independently on multiple model server shards, and merging the results. section 4.3.2 section 4.9
A typical request flow is:
- Divide the original input into meaningful subsets, such as separate portions of a high-resolution video. section 4.3.2
- Assign each subset to a shard using a sharding function similar to hashing or load-balancing algorithms. section 4.3.2
- Process subsets independently, preserve their results, and merge them into the response for the original request. section 4.3.2
Tagging a high-resolution video
A video containing a dog and a kid is divided into two videos. Separate model server shards process the portions independently, then merge the detected entities into one response containing both dog and kid. section 4.3.2
Use the pattern when request data exceeds one machine’s capacity. Unlike replicated services for stateless scaling, sharded services generally maintain state across sub-requests and must combine partial results. Poor partitioning can create meaningless subsets or incomplete entities, requiring cleanup after merging. section 4.3.3
Figure 4.10 An example architecture of the sharded services pattern where a high-resolution video gets divided into two separate videos. Each video represents a subset of the original large request and is processed by different model server shard independently.
Use the architecture figure to trace how one high-resolution video becomes separate shard requests and returns as one combined inference result. section 4.3.2
