Data parallel is a distributed-training strategy for speeding training on a large dataset. It keeps a replica of the same model on each CPU or GPU, splits training data across those devices, and lets every device train on its own subset. It is usually the first distributed option to consider when the model fits on one GPU but training takes too long. section 4.1 section 4.4.3
In synchronous data parallel training, a global batch is divided into per-device minibatches. Workers run training steps simultaneously, aggregate their gradients, then receive the same aggregated gradients to update local model replicas. The replicas therefore remain synchronized at each iteration. Compared with single-device training, the added work is batch partitioning and gradient synchronization; all-reduce is a common aggregation algorithm. section 4.2.1
Figure 4.1 A synchronous data parallelism concept graph. (a) Deep learning training on a single GPU. (b) Synchronous data parallel training with three GPUs.
Compare one-GPU training with three-GPU synchronous training: each GPU handles a minibatch, while gradient aggregation keeps their model updates aligned. section 4.2.1
Data parallel is practical because it preserves the model architecture, but its distributed workflow has operational requirements:
- Prefer it over model or pipeline parallelism when the model fits on one GPU: converting a single-device program generally needs minimal, standardized changes rather than splitting the network into subnetworks. section 4.1
- Register every training process into a group before training. PyTorch uses a world size, process rank, and master address and port; rank 0 is the master process. section 4.2.3
- Plan for communication limits. At every iteration, workers exchange gradients or parameters, and GPU-memory and network transfer can eventually outweigh the speedup from added workers. section 4.2.2
- Persist training state remotely so a failed or slow worker can restart from its most recent state instead of losing the whole group’s progress. section 4.2.2
- Use the master worker for synchronized-job progress and status, and have only that worker persist final models and checkpoints to avoid duplicate artifacts. section 4.5
Framework integration patterns
PyTorch DistributedDataParallel wraps a model and synchronizes gradients during backward computation after processes join the group. TensorFlow uses TF_CONFIG to describe workers and MultiWorkerMirroredStrategy to manage synchronous training. Horovod initializes its runtime, wraps a TensorFlow gradient tape or PyTorch optimizer for averaging, and broadcasts initial state from rank 0. section 4.2.3
A training service can run distributed code across a machine group while treating framework-specific training code as a black box after setup configuration is encapsulated. Kubernetes can reduce cluster resource-management complexity. Horovod is especially useful where one execution approach must support training code written with PyTorch, TensorFlow, MXNet, or PySpark. section 4.0 section 4.5
