A dataset is a managed unit of collected, related data. In a deep learning system, it provides the data that teams explore, label, transform, and use to train models; data quality is central because poor inputs undermine learning. section 1.2.2 section 1.1.1
Datasets begin during data exploration, when data scientists and data engineers collect public or internal data, clean it as needed, and label it. Early work may be informal—scripts, manual copies, or notebook-based analysis—but relevant data improves the likelihood of effective, efficient models. section 1.1.1
A dataset manager organizes the data flow among system components:
- Data collectors create or update datasets by pushing raw data. section 1.2.2
- Workflow orchestration pulls data for processing or format transformation, then returns enhanced training data. section 1.2.2
- Notebooks pull data for exploration and examination, while model training services pull training data. section 1.2.2
A dataset can be dynamic for ingestion and static for reproducible training. Commits record newly ingested data and may carry tags and messages. Preparing training data selects commits and produces a versioned snapshot; its version can later fetch the identical training data. section 2.2.5 section 10.1.1
Figure 2.13 An internal dataset storage overview. A dataset stores two types of data: commits for the ingested raw data and versioned snapshots for the training dataset. The dataset metadata and data URLs are stored in the dataset management service, and the actual data is stored in the cloud object storage service.
The storage layout separates mutable commits from versioned training snapshots, while keeping metadata and data URLs in the management service and files in object storage. section 2.2.5
Reproducible visual-recognition experiments
A researcher uses ImageNet from a dataset management service in JupyterLab, trains and tests several models while changing hyperparameters, and keeps the input fixed with a versioned dataset. Git tracks the corresponding code version, allowing each experimental model to be traced to data and code. section 10.1.1
