A strongly typed dataset defines an explicit schema and data types for records, stabilizing raw-data collection and training-data consumption. It protects training code when dataset formats evolve, while retaining a versioned representation for reproducible fetching. section 2.2.2 section 2.2.7
A production dataset management service organizes the type around distinct responsibilities:
- Accept typed ingestion payloads through a data ingestion API and store each update as a commit. section 2.2.2 section 2.4
- Expose a defined training-data format through a dataset fetching API and store generated training data as a versioned snapshot. section 2.2.2 section 2.4
- Translate between ingestion and fetching schemas with a dataset transformer, rather than coupling collection data directly to training consumption. section 2.2.7
IMAGE_CLASS dataset
An IMAGE_CLASS ingestion archive uses one directory per label, such as cat and dog, with images directly inside each directory. The fetching API emits a zip containing examples.csv, labels.csv, and an examples folder. ImageClassTransformer.ingest() converts incoming archives into commits; compress() filters and merges commits into one training snapshot. section 2.2.7
Figure 2.6 System overview of the sample dataset management service. The sample service contains three main components, a data ingestion API, internal storage, and a dataset fetching API.
Use the system overview to trace how typed data moves between the ingestion API, internal storage, and dataset fetching API. section 2.2.2
