In this chapter, we will discuss the last but critical piece of a deep learning system: workflow orchestration—a service that manages, executes, and monitors workflow automation. Workflow is an abstract and broad concept; it is essentially a sequence of operations that are part of some larger task. If you can devise a plan with a set of tasks to complete a work, this plan is a workflow. For example, we can define a sequential workflow for training a machine learning (ML) model. This workflow can be composed of the following tasks: fetching raw data, rebuilding the training dataset, training the model, evaluating the model, and deploying the model.
Because a workflow is an execution plan, it can be performed manually. For instance, a data scientist can manually complete the tasks of the model training workflow we just described. For example, to complete the “fetching raw data” task, the data scientist can craft web requests and send them to the dataset management (DM) service to fetch a dataset—all with no help from the engineers.