Backward compatibility keeps existing dataset consumers and projects working when upstream data collection or schemas change. In deep learning workflows, unmanaged schema growth can break downstream model training and slow development. A dataset management service separates data developer and data scientist workflows, centralizes schemas, and protects established consumers from incompatible updates. section 2.1.1
Implement compatibility through explicit dataset-management boundaries:
- Define strongly typed schemas for both ingestion and training data, then perform transformation inside the dataset management service. section 2.1.1 section 2.1.2
- Enforce schemas so upstream collection changes cannot unexpectedly affect downstream model-training code or other clients. section 2.1.2
- Version schemas when necessary, but use a new dataset type and schema for changes that would otherwise break existing consumers. section 2.1.2
- Expose a consistent API while the service handles scaling internally, keeping client integration stable across dataset sizes. section 2.1.2
Figure 2.4 A dataset management component creates a good separation between training data collection and consumption by defining strongly typed schemas for both, which allows data development and model algorithm development to iterate in their own loop, thus expediting the project development.
Shows how separate strongly typed ingestion and training schemas isolate data collection from model consumption while enabling independent iteration. section 2.1.1
