A model server shard is a partition of one model-serving instance that processes a subset of a large inference request. Sharding addresses requests too large for one serving component by dividing the input, processing parts independently, and combining the resulting predictions. section 4.3.2
A sharded serving workflow follows these steps:
- Divide the original high-resolution input into separate sub-requests, such as video regions. section 4.3.2
- Send each sub-request to a different model server shard for independent processing. section 4.3.2
- Have each shard retrieve the previously trained model from model storage and generate inferences for its assigned subset. section 4.3.2
- Merge the shard results to produce the inference result for the original request. section 4.3.2
High-resolution video tagging
A video containing a dog and a kid is divided into two videos. One shard processes the dog region and another processes the kid region; merging their entity-tagging results returns dog and kid for the original video. section 4.3.2
Figure 4.10 An example architecture of the sharded services pattern where a high-resolution video gets divided into two separate videos. Each video represents a subset of the original large request and is processed by different model server shard independently.
Use the architecture diagram to trace input partitioning, independent shard processing, and result assembly. section 4.3.2
