DistributedDataParallel (DDP) trains replicated PyTorch models synchronously across processes or GPUs. Each process owns an identical model and optimizer, consumes a distinct data shard, computes locally, and participates in gradient averaging so every replica applies the same update. Unlike centralized DataParallel, DDP removes the coordinating GPU and scales through peer-to-peer communication. section 6.4.3 section 6.4.5
Figure 6.3 The workflow of Data Parallelism. A large data batch is sharded into mini-batches, and each GPU processes one mini-batch using its local copy of the model.

Visualizes batch sharding, replicated models, local computation, and cross-GPU gradient aggregation. section 6.4.3
A practical DDP setup follows these steps:
- Launch one process per GPU, commonly with torchrun, and assign each process a rank and local device. section 6.4.6
- Initialize the distributed process group to establish communication among participating processes. section 6.4.6
- Wrap the device-local model with DistributedDataParallel so backward passes trigger gradient synchronization. section 6.4.6
- Use DistributedSampler to give each process a unique dataset shard while retaining an otherwise familiar training loop. section 6.4.6
Four-process launch
torchrun --nproc_per_node=4 train.py --args launches train.py in four processes, one for each available GPU.
Listing 6.27 Key DDP setup steps in PyTorch
import torch.distributed as dist from torch.nn.parallel import DistributedDataParallel as DDP from torch.utils.data.distributed import DistributedSampler # 1. Initialize the process group dist.init_process_group(backend='nccl') #A # 2. Assign the device for the current process local_rank = int(os.environ['LOCAL_RANK']) torch.cuda.set_device(local_rank) # 3. Wrap the model model = MiniDeepSeekM().to(local_rank) ddp_model = DDP(model, device_ids=[local_rank]) #B # 4. Use the distributed sampler for the dataloader train_sampler = DistributedSampler(train_dataset) train_dataloader = DataLoader(train_dataset, batch_size=..., sampler=train_sampler) #C
Provides the compact PyTorch initialization pattern: process group, local device, DDP wrapper, and distributed sampler. section 6.4.6


