A pipeline is an ordered sequence of steps that moves data or work from one stage to the next. In tabular machine learning, it can mean data preparation, repeatable preprocessing for training and prediction, or automation of the full model lifecycle. The intended meaning matters because these pipeline types solve different problems. section 2.3 section 11.1.1
Use the term precisely:
- A data pipeline cleans or resolves input-data problems, including missing values and schema issues. section 11.1.1
- A training/inference pipeline applies the same transformations during model training and later prediction, keeping categorical or text processing consistent. section 11.1.1
- An ML pipeline automates lifecycle steps such as training, deployment, and monitoring, so a model can be retrained and redeployed consistently. section 11.1.1
Start pipeline design after defining the project goal and locating a target and predictors. Data understanding sits between obtaining data and engineering it; exploration then guides which columns need filtering, special handling, or feature-specific transformations. section 2.3 section 4.2
A compact modeling pipeline
For a binary classification problem, combine feature standardization with logistic regression in a Scikit-learn Pipeline. Evaluate the complete pipeline with cross-validation, then refit it on all available data when performance is suitable. Treat the procedure—not the particular model—as the reusable pattern. section 4.1
For mixed tabular features, separate categorical, numeric, and binary columns before assembling transformations. One-hot, ordinal, and target encoding can be assigned to different categorical columns, while numeric columns may pass through unchanged. Encoding should fit the model: ordinal encoding can suit tree-based models, whereas linear models may need one-hot or target encoding. section 7.2.2
Prefer model-specific pipelines when evidence shows their performance benefit exceeds the cost of maintaining them. A single general pipeline is simpler but may compromise performance because models handle feature types differently. Validation strategy, EDA, and the data pipeline should be checked before modeling. section 7.2.2 section 7.4
Listing 4.1 Example using Scikit-learn for a classification problem
import numpy as np
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_validate
from sklearn.pipeline import Pipeline
X, y = make_classification(n_features=32,
n_redundant=0,
n_informative=24,
random_state=1,
n_clusters_per_class=1
) #1
model = LogisticRegression() #2
pipeline = Pipeline(
[('processing', StandardScaler()),
('modeling', model)]) #3
cv_scores = cross_validate(estimator=pipeline,
X=X,
y=y,
scoring="accuracy",
cv=5) #4
mean_cv = np.mean(cv_scores['test_score'])
std_cv = np.std(cv_scores['test_score'])
print(f"accuracy: {mean_cv:0.3f} ({std_cv:0.3f})") #5
model.fit(X, y)
Inspect a runnable Scikit-learn workflow that creates data, standardizes features, fits logistic regression, and evaluates the combined pipeline with cross-validation. section 4.1
At production scale, an ML pipeline replaces repeated manual notebook and deployment work with a coded unit. One implementation starts with a cleaned dataset, splits it into train, validation, and test files in Cloud Storage, runs a Python training script in a container, and deploys the resulting model to a Vertex AI endpoint. section 11.1 section 11.3.1
Figure 11.1 Three kinds of pipelines and how they relate
Compare how data, training/inference, and ML pipelines occupy different parts of an end-to-end machine-learning workflow. section 11.1.1
