chapter five

5 Workflow patterns

 

This chapter covers

  • Using workflows to connect different machine learning system components (data ingestion, distributed model training, and model serving).
  • Composing complex but maintainable structures within machine learning workflows with the fan-in and fan-out patterns.
  • Accelerating machine learning workloads with concurrent steps that leverages the synchronous and asynchronous patterns.
  • Improving performance and avoiding duplicate workloads in the workflows with the help of the step memoization pattern.

5.1 What is workflow?

5.2 Fan-in and fan-out patterns: Composing complex machine learning workflows

5.2.1 Problem

5.2.2 Solution

5.2.3 Discussion

5.2.4 Exercises

5.3 Synchronous and asynchronous patterns: Accelerating workflows with concurrency

5.3.1 Problem

5.3.2 Solution

5.3.3 Discussion

5.3.4 Exercises

5.4 Step memoization pattern: Skipping redundant workloads via memoized steps

5.4.1 Problem

5.4.2 Solution

5.4.3 Discussion

5.4.4 Exercises

5.5 References

5.6 Summary