Provenance records where data comes from, how transformations change it, and which dependencies produced an artifact. It supports auditability, rollback, reproducibility, and troubleshooting across data and model workflows. section 2.3.2 section 8.1
A practical lineage architecture separates artifact storage from metadata. Files such as datasets, models, code, and processing outputs reside in file or object storage, while a metadata store records URLs, execution identifiers, models, and related inputs and outputs. section 8.1
A production workflow assigns provenance responsibilities across several components:
- Version-control datasets and pipeline definitions so each update, transformation, and dependency can be inspected or reproduced. section 2.3.2
- Associate training metrics with the training dataset and code, enabling complete lineage for experimental models. section 10.1.3
- Retain metadata alongside artifacts to support performance comparisons, reproducibility, and troubleshooting. section 8.1
Image processing with Pachyderm
An edges pipeline watches an images dataset. When a new image is committed, Pachyderm launches a Docker-based OpenCV job that writes an edge image to the edges dataset. Each resulting commit records its parent, pipeline specification, and input images commit, allowing the transformation path to be inspected. section 2.3.2
Figure 2.18 The Pachyderm platform runs with two kinds of objects—a pipeline and versioned data. The pipeline is the computational component, and the data is the version-control primitive. A data change in the “raw dataset” can trigger a pipeline job to process the new data and save the result to the “mature dataset.”
Use the pipeline-and-versioned-data architecture to trace how a raw-dataset change triggers processing and produces a mature dataset. section 2.3.2
