chapter nine
9 Model serving in practice
This chapter covers
- Building a sample predictor with the model service approach
- Building a sample service with TorchServe and the model server approach
- Touring popular open source model serving libraries and systems
- Explaining production model release process
- Discussing post production model monitoring
In the previous chapter, we discussed the concept of model serving, as well as user scenarios and design patterns. In this chapter, we will focus on the actual implementation for these concepts in production.
As we’ve said, one of the challenges to implementing model serving nowadays is that we have too many possible ways of doing it. In addition to multiple blackbox solutions there are also many options for customizing and building all or part of it from scratch. We think the best way to teach you the intuition of choosing the right approach is with concrete examples.