A versioned snapshot is an immutable training-data aggregate created from selected dataset commits. It gives a changing dataset a static, reproducible form: new ingestion creates new commits, while an existing snapshot continues to identify the same training data for future retrieval. section 2.2.2 section 2.2.5
Snapshots are created by the training-data preparation workflow rather than by ingestion. A preparation request identifies a dataset and filters commits by tags; the service selects matching commits, hashes their ordered ID list, and records a versionedSnapshot in a version-hash registry. The returned hash is both the snapshot identifier and the handle for later status checks and download retrieval. section 2.2.4 section 2.2.4
Figure 2.11 A high-level overview of the eight steps to responding to a dataset build request: (1) the user submits a dataset preparation request with data filters; (2) DM selects data from commits that satisfy the data filters; (3 and 4) DM generates a version string to represent the training data; and (5–8) DM starts a background job to produce the training data.
This workflow shows the separation between selecting commits, assigning a version string, and asynchronously producing the training-data artifact. section 2.2.4
A versioned snapshot supports several production requirements:
- Reproducibility: a version hash can retrieve the same training data at any future time, even as later commits extend the dataset. section 2.2.4 section 2.2.8
- Asynchronous preparation: the service can return a stable snapshot ID immediately, build files in a background job, and expose a download URL when ready. section 2.2.4
- Caching: an already registered hash returns the existing snapshot instead of rebuilding identical selected commits, saving computation and bandwidth. section 2.2.4
- Large-data API consistency: preparation and status/query operations avoid making request behavior depend on dataset size. section 2.2.4 section 2.2.8
The snapshot is metadata plus generated data location, not necessarily the data files themselves. Dataset management tracks commits, snapshots, version hashes, and URLs, while commit and snapshot files can reside in cloud object storage. This keeps the service focused on organization, ownership, audit-related metadata, and training-data versions. section 2.2.5
Figure 2.13 An internal dataset storage overview. A dataset stores two types of data: commits for the ingested raw data and versioned snapshots for the training dataset. The dataset metadata and data URLs are stored in the dataset management service, and the actual data is stored in the cloud object storage service.
The storage view distinguishes dynamic ingested commits from static training snapshots and separates service metadata from object-stored files. section 2.2.5
Preparing and fetching a repeatable training dataset
Submit PrepareTrainingDataset with a dataset ID and a tag such as DataType=Training. The service selects commits carrying that tag, derives a hash from the selected commit IDs, and starts building the aggregate. Save the returned hash with the training run. Later, call FetchTrainingDataset with the dataset ID and hash; the registry returns the corresponding snapshot and, after preparation completes, its downloadable URL. Filtering is at commit level, not individual-example level. section 2.2.4 section 2.2.4
Immutability is the boundary that makes a snapshot dependable: each dataset update produces a new commit, and each preparation request creates or reuses a versioned snapshot. Snapshots and commits persist without expiration in this design. Operations such as deletion, reverting commits, and audit-history management are identified as omitted extensions rather than part of the core snapshot flow. section 2.2.8 section 2.2.8
