9 Machine Learning With the Full Stack
This chapter covers:
- Developing a custom framework that makes it easier to develop models and features for a particular use case.
- Training a deep learning model in a workflow.
- Summarizing the lessons learned in this book.
We have now covered all layers of the infrastructure stack, shown again in Figure 9.1, except the topmost one: Model Development. Also we only scratched the surface of the Feature Engineering layer in Chapter 7. Isn’t it paradoxical that a book about machine learning and data science infrastructure spends so little time talking about the core concerns of machine learning: models and features?
Figure 9.1. Infrastructure stack with the data scientist’s areas of interest
The focus is deliberate. First, many excellent books already exist about these topics. Also mature modeling libraries like TensorFlow and PyTorch come with a plethora of in-depth documentation and examples. Second, as depicted in Figure 9.1, these topics tend to be the core areas of expertise of professional data scientists and machine learning engineers whereas the lower layers are not. To boost the day-to-day productivity of data scientists effectively, it makes sense to help them where help is needed the most: the lower layers of the stack.