A model trained once in a notebook may work, yet be hard to reproduce when new data arrives. A pipeline turns the preparation, training, checks, and outputs into a repeatable workflow rather than a collection of manual actions. section 10.1
A pipeline consists of steps with inputs and outputs: one step’s output can feed another. Its dependencies can be represented as a directed acyclic graph (DAG). In machine learning, the scope matters. A training pipeline takes a dataset and possibly metadata and produces a trained model; an inference pipeline takes raw data and produces predictions. A model with feature scaling is itself a smaller pipeline, not the whole training workflow. section 10.1.1
Figure 10.1 A DAG scheme representing a training pipeline
A DAG view helps distinguish the dependencies among training steps from the model inside them. section 10.1.1
For a training pipeline, work through the lifecycle and its operating needs:
- Fetch and preprocess data, train and evaluate the model, then test, postprocess, package artifacts, and generate a report. section 10.7
- Resolve step dependencies, retain artifacts, and make runs reproducible from a pipeline version and parameters. Python scripts joined by shell scripts can suffice; an ML-specific platform can simplify orchestration and maintenance. section 10.2
- Choose configurability for the experiments and tests you need, without making the workflow needlessly complex. If data outgrows one machine, weigh simpler but limited vertical scaling against distributed horizontal scaling. section 10.3 section 10.7
- Test important components with unit tests and run a fast whole-pipeline smoke test on reduced data or training settings. A full training run may take days, so it is not a practical check after every change. section 10.5
Demand forecasting
A store-level demand forecast needs cleaning, feature engineering, normalization, and a time-aware training–validation split before comparing model choices. Keeping these steps in the pipeline makes experiments and maintenance more manageable. section 10.6.1
Image–text search
A search-ranking pipeline can fine-tune image and text encoders, save both models and their inference parameters, and produce training metrics. Starting on one GPU while allowing for multiple GPUs avoids committing to distributed training before it is needed. section 10.6.2
Inference has its own operating constraints: preprocessing and feature computation precede predictions, and postprocessing may follow them. Decide whether to share a framework with training or convert the model for serving; either choice should reflect production needs rather than training convenience alone. section 15.0 section 15.3.1
