A data pipeline automates repeated movement and processing of data across stages. In production machine learning, it pulls from data sources, keeps datasets updated and versioned, and prepares data through transformation, enrichment, and labeling before training or downstream use. section 1.1.1 section 1.1.2
Pipelines are part of a wider production workflow: datasets feed repeatable training and evaluation, approved models move through deployment, and monitoring assesses data, models, and serving over time. Infrastructure and workflow orchestration reduce handoffs so data scientists can set up pipelines alongside deployment and monitoring. section 1.1.1 section 1.1.4
A version-controlled pipeline platform can separate responsibilities into:
- Versioned data repositories, where creates, writes, and deletes become commits and actual files reside in cloud object storage. section 2.3.2
- Pipeline definitions that chain user-supplied Docker images for scraping, ingestion, cleaning, munging, wrangling, or other transformations. section 2.3.2
- Scheduled jobs that run and scale when input data changes, producing output datasets with recorded provenance. section 2.3.2
Image edge-processing pipeline
An edges pipeline watches an images dataset. Adding an image launches a job using a pachyderm/opencv container to parse it and save an edge image to the edges dataset. The resulting commits can then be inspected for timestamps, parents, and provenance. section 2.3.2
Figure 2.18 The Pachyderm platform runs with two kinds of objects—a pipeline and versioned data. The pipeline is the computational component, and the data is the version-control primitive. A data change in the “raw dataset” can trigger a pipeline job to process the new data and save the result to the “mature dataset.”
The diagram shows the core event flow: a change in raw, versioned data triggers computation and writes a mature output dataset. section 2.3.2
For labeled-image workflows, the same event-driven pattern can start model training automatically when new labels arrive. This couples data updates to repeatable training while preserving a traceable path from inputs to outputs. section 1.1.1 section 2.3.2
