A data contract is an explicit pair of schemas that defines the handshake between data producers and consumers. In a multi-team deep learning workflow, it prevents downstream users from having to infer how upstream data should be read, reducing coordination pressure as datasets and projects multiply. section 2.1.1
A dataset management service can enforce the contract at two boundaries: an ingestion schema validates incoming raw data, while a training-data schema converts and guarantees the format delivered to model development. Commits and versioned snapshots retain that output-schema guarantee when assembling a training dataset. section 2.2.6
This separation supports independent iteration:
- Data collection must be converted into the required ingestion format before entering dataset management. section 2.2.6
- Training consumers receive data conforming to the training schema despite changes in collection methods. section 2.2.6
- Strongly typed schemas separate collection from consumption, so data and model work can iterate in their own loops. section 2.1.1
TEXT_INTENT contract
A TEXT_INTENT ingestion contract accepts CSV input with a text utterance in the first column and one or more labels afterward. Its training contract produces a ZIP containing examples.csv, which maps utterances to label IDs, and labels.csv, which maps label IDs to label names. Using this self-defined structure avoids binding the dataset interface to PyTorch or TensorFlow formats. section 2.2.6
Figure 2.15 Each type of dataset has two data schemas: ingestion data schema and training data schema. These two schemas will ensure that the data we accept and the data we produce follow our data spec.
The two-schema boundary shows the contract's accepted input and guaranteed training output. section 2.2.6
