A training data schema defines the data a training process is allowed to consume. In a dataset management service, it is paired with an ingestion schema: ingestion validates raw input, while the training schema converts and preserves output in a stable training format. This contract separates data collection changes from model training, letting data developers and data scientists iterate independently. section 2.1.1 section 2.2.6
For a strongly typed dataset, the service validates incoming data against the ingestion schema, converts it to the training schema, saves converted data as commits, and merges selected commits into a versioned training snapshot. Consumers can therefore rely on the output format even when collection methods change upstream. section 2.2.6
A training schema serves several production needs:
- Stabilize the interface between data collection and model training through a strongly typed output contract. section 2.1.1 section 2.2.6
- Keep training data independent of particular model-training frameworks by using a self-defined structure rather than PyTorch or TensorFlow dataset formats. section 2.2.6
- Promote an experimental schema-free dataset to a typed dataset when its format needs protection from later updates. section 2.1.1 section 2.2.7
TEXT_INTENT output format
A TEXT_INTENT training dataset is a zip file containing examples.csv and labels.csv. examples.csv maps each text utterance to one or more label IDs; labels.csv maps those IDs to label names. For example, “I am still waiting on my credit card” maps to label IDs 0 and 1, which map to activate_my_card and card_arrival. section 2.2.6
Figure 2.15 Each type of dataset has two data schemas: ingestion data schema and training data schema. These two schemas will ensure that the data we accept and the data we produce follow our data spec.
The two-schema architecture shows the training schema as the controlled output side of a dataset contract, distinct from raw-data ingestion. section 2.2.6
Adding IMAGE_CLASS
To formalize image classification training data, define a zip output with examples.csv, labels.csv, and an examples folder of image files. Then define the ingestion format and implement a DatasetTransformer that parses input, writes converted commits, and merges matching commits into a training snapshot. section 2.2.7
