A dataset is a collection of records or examples consumed to train or evaluate a machine-learning model. Its size and arrival rate shape the system architecture: a dataset that grows hourly can outlive a laptop’s fixed storage and compute capacity. section 1.1.1
When the entire dataset cannot fit on one machine, partition it across machines with sufficient disk storage. Training can consume partitions sequentially, transferring the model between machines, or use workers that process partitions in parallel. Parallel processing reduces waiting but introduces coordination questions about competing gradient updates, data quality, and model quality. section 1.1.2
Treat dataset distribution as an operational design choice:
- Use partitioning to store data that exceeds a single machine’s capacity. section 1.1.2
- Use sequential consumption when simplicity matters, while recognizing that large partitions can create long waits. section 1.1.2
- Use parallel workers to accelerate training, then assess synchronization conflicts and uneven partition quality. section 1.1.2
- Plan for unusable or corrupted data and unstable worker or network infrastructure in distributed training. section 3.0
A 30 GB dataset
Divide a 30 GB dataset into three 10 GB partitions, placing each partition on a separate machine. A sequential workflow trains on the first partition, transfers the model to the second machine, and continues. A parallel workflow assigns one worker to each partition, trading shorter training time for coordination and gradient-update decisions. section 1.1.2
Figure 1.1 An example of dividing a large dataset into three partitions on three separate machines that have sufficient disk storage
Visualizes a large dataset divided into three partitions across separate machines, making distributed storage and consumption concrete. section 1.1.2
