10 Training pipelines
This chapter covers
- The essence of training pipelines
- Tools and platforms you can use to build and maintain training pipelines
- Scalability and configurability of training pipelines
- Methods of testing pipelines
There’s an empirical heuristic to distinguish experienced machine learning (ML) engineers from newcomers: ask them to describe a working system’s training procedure in one sentence. Newcomers tend to focus on models, while somewhat experienced individuals include data processing. Mature engineers often describe the pipeline—a list of stages required to produce a trained ML model in the end. In this chapter, we will walk in ML engineers’ shoes to analyze these steps and discuss how to interconnect and orchestrate them.