A schema is an explicit contract for the shape and types of data exchanged between producers and consumers. In dataset management, strongly typed schemas validate incoming data and guarantee that generated training data follows an agreed specification, preventing upstream collection changes from unexpectedly breaking training code. section 2.1.2 section 2.1.1
Define separate schemas at the ingestion and training boundaries. The ingestion schema states what raw data a service accepts; the training schema states the stable format it emits. This separation lets data collection and model development iterate independently while preserving a contract at each boundary. section 2.2.6 section 2.1.1
Use schemas to make dataset changes operationally safe:
- Validate raw inputs during ingestion, then transform them into the declared training format before saving a commit. section 2.2.6
- Prefer backward-compatible schema updates; when a requirement is breaking, create a new dataset type and schema rather than alter the existing contract. section 2.1.2
- Keep public ingestion and fetching APIs consistent across dataset types, while allowing internal parsing and storage formats to change. section 2.1.2
- Start experimental work with an unvalidated generic dataset, then promote it to a strongly typed dataset when its requirements mature. section 2.4
Text intent dataset contract
A TEXT_INTENT ingestion schema accepts a CSV whose first column is a text utterance and whose remaining columns are labels. Its training schema emits a zip containing examples.csv, which maps utterances to label IDs, and labels.csv, which maps each ID to a label name. The self-defined format keeps training data independent of PyTorch and TensorFlow dataset formats. section 2.2.6
Adding an image classification schema
For IMAGE_CLASS, define the training output first: a zip with image-to-label manifests and an examples directory. Then define ingestion: a zip of label-named directories containing images. Implement a DatasetTransformer to parse the ingestion format, create commits, and merge selected commits into a training snapshot. section 2.2.7
Figure 2.15 Each type of dataset has two data schemas: ingestion data schema and training data schema. These two schemas will ensure that the data we accept and the data we produce follow our data spec.
The diagram shows the two-schema boundary: one contract governs accepted inputs, and another governs produced training data. section 2.2.6
