Pachyderm is a Kubernetes-based platform for version-controlled, automated data pipelines backed by cloud object storage. It separates versioned data from computation: repositories hold commits, files, and branches, while pipelines run containerized transformations. section 2.3.2
Figure 2.18 The Pachyderm platform runs with two kinds of objects—a pipeline and versioned data. The pipeline is the computational component, and the data is the version-control primitive. A data change in the “raw dataset” can trigger a pipeline job to process the new data and save the result to the “mature dataset.”
Follow the event-driven workflow from a change in a raw dataset to a pipeline job and a transformed mature dataset. section 2.3.2
A Pachyderm deployment assigns distinct responsibilities to its core objects:
- Repositories provide Git-style dataset versioning through commits, files, and branches; Pachyderm stores metadata internally and files in object storage. section 2.3.2
- Pipelines chain user-defined Docker images for scraping, ingestion, cleaning, munging, wrangling, and other transformations. section 2.3.2
- Jobs execute pipeline code, and Pachyderm schedules, scales, and launches jobs when input data changes. section 2.3.2
- Commits capture dataset and pipeline changes, enabling rollback, audit history, and data-provenance tracing. section 2.3.2
Image edge detection and model training
An `edges` pipeline watches the `images` repository and runs the `pachyderm/opencv` image with `python3 /edges.py`. New images trigger jobs that write edge pictures to `edges`; newly labeled images can likewise trigger automated object-detection model training. section 2.3.2
