A sharding function deterministically maps each request or sub-request to the model-server shard responsible for processing it. It commonly combines a hashing function with the modulo operator, such as hash(request) % 10. section 4.3.2
A useful sharding function provides:
- Stable assignment: the same input always selects the same shard. section 4.3.2
- Uniform distribution: outputs spread requests across the available shards. section 4.3.2
- Scalable request processing: separate sub-requests can run on different model-server shards before their results are merged. section 4.3.2
High-resolution video request
Divide a large video request into two sub-requests, route them to different model-server shards, process them independently, and merge the inference results for entities such as a dog and a kid. section 4.3.2
Figure 4.10 An example architecture of the sharded services pattern where a high-resolution video gets divided into two separate videos. Each video represents a subset of the original large request and is processed by different model server shard independently.
Use the architecture to trace how one high-resolution video becomes independent shard-level requests and returns as one merged result. section 4.3.2
