DistributedDataParallel
DistributedDataParallel is a PyTorch API that trains replicated models across processes or devices by synchronizing gradients during data-parallel computation.
Graph Neural Networks in Action
DistributedDataParallel is a PyTorch API that runs replicated model processes across multiple GPUs or machines while synchronizing gradients.
- How it is used
- Uses DistributedDataParallel to spread training across multiple GPUs, with small-scale clusters also considered.
- How deep it goes
- Walks through process groups, manual data division, model wrapping, training, synchronization, and cleanup, then illustrates a two-GPU script.
- Examples
- Demonstrates MNIST training with a Flatten-and-Linear network and points to a GCN example using Amazon Products.
- What you can do after
- Apply it via PyTorch DistributedDataParallel to run multi-GPU training while dividing data and coordinating processes.
Read how Graph Neural Networks in Action explains DistributedDataParallel →
Designing Deep Learning Systems: A guide for software engineers
PyTorch's DistributedDataParallel library component that runs a model across processes or machines and coordinates distributed updates.
- How it is used
- Runs PyTorch data-parallel training across processes on one machine or across multiple machines.
- How deep it goes
- Explains process registration, model wrapping, gradient synchronization, and bucket tuning without descending into custom synchronization internals.
- Examples
- Uses process-group setup, GPU model wrapping, synchronized backward, and gradient-bucket sizing as implementation examples.
- What you can do after
- Apply it via PyTorch by initializing a process group, wrapping a model with DDP, and training with synchronized gradients.
Build a DeepSeek Model (From Scratch)
nn.DistributedDataParallel is a PyTorch module for synchronously training replicated models across multiple processes or devices.
- How it is used
- Scales synchronous model training across GPUs by combining local computation with peer-to-peer gradient averaging.
- How deep it goes
- Explains process setup, data sharding, backward-pass all-reduce, communication overlap, and synchronized optimizer updates.
- Examples
- Uses a four-GPU torchrun command, a batch-sharding diagram, and a PyTorch setup listing.
- What you can do after
- Apply via PyTorch by launching processes, assigning devices, wrapping models with DDP, and partitioning data with DistributedSampler.
Read how Build a DeepSeek Model (From Scratch) explains DistributedDataParallel →
Also covered in
What to know first
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- backpropagationBackpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
- allreduceAllreduce is a collective communication operation that combines corresponding values from participating processes or devices and distributes the result to every participant.
- collective communicationCollective communication is a coordinated operation in which multiple processes or devices exchange, combine, or synchronize data as part of a distributed computation.
Related concepts
- allreduceuses · Allreduce is a collective communication operation that combines corresponding values from participating processes or devices and distributes the result to every participant.
- data parallelismimplements · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
- GPUuses · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and other data-parallel workloads.
- gradient synchronizationuses · Gradient synchronization is the coordination of gradients computed by distributed training processes so replicated models receive consistent parameter updates.
Questions about DistributedDataParallel
What is DistributedDataParallel?
DistributedDataParallel is a PyTorch API that trains replicated models across processes or devices by synchronizing gradients during data-parallel computation.
Which Manning books explain DistributedDataParallel?
Graph Neural Networks in Action, Designing Deep Learning Systems: A guide for software engineers and Build a DeepSeek Model (From Scratch).
What should I know before learning DistributedDataParallel?
Manning books that teach DistributedDataParallel usually introduce gradient, backpropagation, allreduce and collective communication first.
What is the difference between DistributedDataParallel and DataParallel?
DistributedDataParallel is a PyTorch API that trains replicated models across processes or devices by synchronizing gradients during data-parallel computation. DataParallel is a parallel-computation mechanism that replicates model computation across devices while distributing input data among them.
What is DistributedDataParallel also called?
DistributedDataParallel is also called DDP.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


