Thus far, you have learned about the individual stages or steps of machine learning in isolation. Focusing on one step of machine learning at a time helped to concentrate your effort on a more manageable scope of work. However, to deploy a production machine learning system it is necessary to integrate these steps into a single pipeline: the outputs of a step flowing into the inputs of the subsequent steps of the pipeline. Further, the pipeline should be flexible enough to enable the hyperparameter optimization (HPO) process to manage and to experiment with the specific tasks executed across the stages of the pipeline.
In this chapter, you will learn about the concepts and the tools you can use to integrate the machine learning pipeline, deploy it to AWS, and train a DC Taxi fare estimation machine learning model using experiment management and hyperparameter optimization.