DistributedDataParallel
DistributedDataParallel is a PyTorch module that replicates a model across processes or devices and synchronizes gradients during data-parallel training.
Graph Neural Networks in Action
DistributedDataParallel is a PyTorch API that runs replicated model processes across multiple GPUs or machines while synchronizing gradients.
- How it is used
- Uses DistributedDataParallel to spread training across multiple GPUs, with small-scale clusters also considered.
- How deep it goes
- Walks through process groups, manual data division, model wrapping, training, synchronization, and cleanup, then illustrates a two-GPU script.
- Examples
- Demonstrates MNIST training with a Flatten-and-Linear network and points to a GCN example using Amazon Products.
- What you can do after
- Apply it via PyTorch DistributedDataParallel to run multi-GPU training while dividing data and coordinating processes.
Read how Graph Neural Networks in Action explains DistributedDataParallel →
Designing Deep Learning Systems: A guide for software engineers
PyTorch's DistributedDataParallel library component that runs a model across processes or machines and coordinates distributed updates.
- How it is used
- Runs PyTorch data-parallel training across processes on one machine or across multiple machines.
- How deep it goes
- Explains process registration, model wrapping, gradient synchronization, and bucket tuning without descending into custom synchronization internals.
- Examples
- Uses process-group setup, GPU model wrapping, synchronized backward, and gradient-bucket sizing as implementation examples.
- What you can do after
- Apply it via PyTorch by initializing a process group, wrapping a model with DDP, and training with synchronized gradients.
Related concepts
- allreduceuses · Allreduce is a collective communication operation that combines corresponding values across participating processes or devices and distributes the resulting aggregate to every participant.
- gradient synchronizationuses · Gradient synchronization is the coordination of gradients computed by distributed training processes so replicated models receive consistent parameter updates.
- GPUuses · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and data-parallel workloads.
- data parallelimplements · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.