An ingestion data schema defines the accepted representation of raw dataset input. In a dataset management service, it validates incoming data before transformation into a separate training data schema, preventing collection changes from breaking model training. section 2.1.1 section 2.2.6
For strongly typed dataset types, the ingestion and training schemas form a data contract between data collection and training consumers. Strong typing supports backward-compatible upgrades; a schema-free GENERIC type trades validation and transformation for early-stage flexibility. section 2.1.1 section 2.2.6
The ingestion path follows a defined boundary:
- Convert collected data into the ingestion format required by the dataset type. section 2.2.6
- Validate raw input against the ingestion data schema. section 2.2.6
- Transform validated data into the training format and save it as a commit. section 2.2.6
- Merge selected commits into a versioned training snapshot that still obeys the training schema. section 2.2.6
TEXT_INTENT input
A TEXT_INTENT ingestion schema accepts CSV: the first column is a text utterance and remaining columns are labels. For example, “I am still waiting on my credit card” can carry activate_my_card and card_arrival labels. section 2.2.6
IMAGE_CLASS input
An IMAGE_CLASS ingestion payload is a zip archive whose root contains label-named directories such as cat and dog. Each directory contains only images for that label and no nested directories. section 2.2.7
Figure 2.15 Each type of dataset has two data schemas: ingestion data schema and training data schema. These two schemas will ensure that the data we accept and the data we produce follow our data spec.
This diagram shows the two-schema contract: one schema governs accepted input, while the other governs produced training data. section 2.2.6
