Cloud object storage provides shared file storage for distributed systems. Components read task inputs and write outputs there, while centralized storage lets web servers, schedulers, and workers operate independently. section 9.2.2
A dataset platform can keep metadata, audit history, branches, and provenance internally while storing the actual files in cloud object storage. This separates version-control operations from bulk data persistence. section 2.3.2
Use cloud object storage when a system needs to:
- Share outputs between dependent workflow tasks without coupling workers to one another. section 9.2.2
- Support workers running on different infrastructure, such as Kubernetes or Amazon EC2. section 9.2.2
- Persist files behind versioned datasets while retaining commits, branches, and provenance in a management layer. section 2.3.2
Image processing pipeline
An images dataset receives a new image. A Pachyderm pipeline launches an OpenCV container, reads the dataset, detects edges, and writes the result to an edges dataset. Each update can become a commit, enabling history, rollback, and provenance inspection. section 2.3.2
Figure 2.18 The Pachyderm platform runs with two kinds of objects—a pipeline and versioned data. The pipeline is the computational component, and the data is the version-control primitive. A data change in the “raw dataset” can trigger a pipeline job to process the new data and save the result to the “mature dataset.”
Use the architecture diagram to trace how a raw-data change triggers computation and produces a versioned mature dataset. section 2.3.2
