liveBook · Manning

Machine learning workflow

As explained in Software Engineering for Data Scientists

A machine learning pipeline is an organized sequence of data-processing and modeling steps supporting the development or operation of a machine learning system.

A machine learning workflow organizes the path from data to predictions. It extends a data pipeline with processing, feature creation, feature selection, model training, evaluation, deployment, and ongoing monitoring. The goal is an automated, repeatable, and hardened process rather than a collection of scattered notebooks. section 1.4 section 1.4.6

A practical workflow commonly includes these steps:

Customer churn prediction

A focused pipeline ingests customer churn data, trains a gradient boosting model on training data, predicts outcomes for a separate test dataset, and writes those predictions to a database table. A more complete workflow can then add rigorous tests and automatic scheduling. section 9.2

Pipeline structure also improves maintainability. Separate scripts such as 1_fetch_dataset.py, 2_clean_dataset.py, 3_feature_engineering.py, 4_feature_selection.py, 5_train_model.py, and 6_generate_evaluation_metrics.py make responsibilities visible and support collaboration. Production use additionally requires source control, scaling, readable structure, and exception handling. section 1.4.6 section 1.5

Figure 1.7 A sample workflow of a machine learning pipeline extending from a data pipeline. This diagram shows an outline of a typical machine learning pipeline. Keep in mind that the starting point is data. Data is always needed for any machine learning pipeline. The final part is monitoring the model to validate its performance on an ongoing basis. These and the in-between components are detailed in the following subsections.
ch1 combined diagram

Use the workflow diagram to follow the pipeline from required input data through intermediate processing and modeling steps to ongoing model monitoring. section 1.4

How Software Engineering for Data Scientists covers it

How deep it goes
Traces named workflow components and file organization, while deferring rigorous testing and scheduling to later development.
Examples
Uses customer churn data, feature transformations, gradient boosting, evaluation scripts, database predictions, and monitoring metrics.
What it assumes
Assumes access to data and teaches how to organize its processing and modeling into dependable steps.
What you can do after
Build it from scratch as a sequential workflow that trains models, produces predictions, stores outputs, and supports monitoring.
How it connects
Connects data pipelines, feature engineering, feature selection, model training, evaluation, deployment, monitoring, testing, and scheduling.

Also explained in

Compare how 9 books explain machine learning workflow →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Software Engineering for Data Scientists. The section links point to the parts of the book it draws on.