4 Model serving patterns
This chapter covers
- Using model serving to generate predictions or make inferences on new data with previously trained machine learning models.
- Handling the growing number of model serving requests and achieving horizontal scaling with the help of replicated model serving services.
- Processing large model serving requests by leveraging the sharded services pattern.
- Assessing model serving systems and determining whether event-driven design would be beneficial for improving resource efficiency.
In the previous chapter, we explored some of the challenges involved in the distributed training component. We introduced a couple of practical patterns that can be incorporated into this component. Distributed training is the most critical part of a distributed machine learning system. For example, we’ve seen challenges when training very large machine learning models that tag main themes in new YouTube videos but cannot fit in a single machine and how we can overcome the difficulty using the parameter server pattern. We also learned how to leverage the collective communication pattern to speed up distributed training for smaller models and avoid the unnecessary communication overhead among parameter servers and workers. Last but not least, we’ve talked about some of the vulnerabilities that are often seen in distributed machine learning systems due to corrupted datasets, unstable networks, and preempted worker machines and how we can address those issues.