A commit is the immutable record created when a dataset is created or updated. It captures a newly ingested batch, its data location, and annotations such as tags and a message. Keeping each update separate avoids irreversible in-place changes and preserves the inputs needed to reproduce earlier training data. section 2.2.3 section 2.2.8
Commits make a dataset dynamic during collection while leaving training inputs selectable and stable. The ingestion API appends each new commit to the dataset’s commit list; the fetching API later filters those commits, commonly by tags, and converts the selected set into a versioned snapshot. section 2.2.4 section 2.2.5
A practical commit workflow is:
- Upload new data to object storage and submit a CreateDataset or UpdateDataset request with its data URL. section 2.2.3
- Assign a new commit ID, ingest and transform the data, then append the commit metadata to the dataset. section 2.2.3
- Attach tags and an optional message so later requests can track history and filter data changes. section 2.2.3
- Select commits for a training request and derive a version hash from the selected commit IDs. section 2.2.4
The service stores commit and snapshot files in cloud object storage while retaining dataset metadata and file links in the dataset-management layer. This separates large-file storage from organization, audit history, statistics, ownership, and snapshot tracking. section 2.2.5
Correcting a bad ingestion
If a data engineer uploads mislabeled data, the update remains an identifiable commit rather than silently altering the current dataset. A management API could remove or soft-delete that commit; training requests can also exclude it by selecting commits with appropriate tags. section 2.2.3 section 2.2.4
This pattern also appears in Pachyderm: every create, write, or delete is treated as a commit, with change history and data provenance available for inspection or rollback. Repositories retain metadata internally while actual files reside in cloud object storage. section 2.3.2
Figure 2.13 An internal dataset storage overview. A dataset stores two types of data: commits for the ingested raw data and versioned snapshots for the training dataset. The dataset metadata and data URLs are stored in the dataset management service, and the actual data is stored in the cloud object storage service.
The storage view distinguishes raw ingested commits from versioned training snapshots and shows where metadata, URLs, and files reside. section 2.2.5
