A dataset manager is the data-management component of a deep learning system. It collects, organizes, describes, and stores data as datasets: related units of collected data. Its central purpose is to make data available for exploration, labeling, and model training while protecting training quality from garbage-in, garbage-out failures. section 1.2.2
In a reference architecture, the dataset manager is a shared system component rather than a private file location for one training job. It provides training input, supports exploration activities, and tracks lineage between datasets. section 1.4
Its main interactions connect the data lifecycle:
- Data collectors push raw data to create or update datasets. section 1.2.2
- Workflow orchestration pulls data for processing, enhancement, or format transformation, then pushes resulting data back. section 1.2.2
- Data scientists, researchers, and data engineers pull data into Jupyter Notebook for exploration and examination. section 1.2.2
- The model training service pulls training data for model training. section 1.2.2
Figure 1.3 An overview of a typical deep learning system that includes basic components to support a deep learning development cycle. This reference architecture can be used as a starting point and further tailored. In later chapters, we discuss each component in detail and explain how it fits into this big picture.
The reference architecture places the dataset manager beside model training, model serving, workflow orchestration, metadata and artifacts storage, and an interactive data science environment. The components are assumed to be networked microservices, while the architecture remains a starting point to tailor for a use case. section 1.2.1
