Torch.distributed
torch.distributed is PyTorch’s package for initializing processes and coordinating communication in distributed computation across devices or machines.
Deep Learning with PyTorch, Second Edition
PyTorch’s distributed-training submodule for coordinating computation and communication across processes, devices, or machines.
- How it is used
- Coordinates training across multiple processes, GPUs, or machines when one GPU is insufficient.
- How deep it goes
- Works through process terminology and initialization with spawning, TCPStore, and torchrun, then connects the APIs to parallel training strategies.
- Examples
- Shows four-process spawning, a TCPStore-backed process group, and a torchrun-launched script.
- What you can do after
- Build it from scratch with process-group initialization, or apply it via torchrun to launch a distributed training script.
Read how Deep Learning with PyTorch, Second Edition explains torch.distributed →
Graph Neural Networks in Action
torch.distributed is the PyTorch package for initializing processes and coordinating communication in distributed computation.
- How it is used
- Coordinates multi-process PyTorch training across GPUs or machines through process initialization, collective communication, and synchronization.
- How deep it goes
- Explains the workflow, synchronization points, launch pattern, and NCCL or Gloo backends without detailing lower-level communication APIs.
- Examples
- Demonstrates MNIST training and points to a GCN using Amazon Products, NeighborLoader, and a dual-GPU environment.
- What you can do after
- Operate/configure distributed training with DDP, NCCL or Gloo, process spawning, device assignment, and synchronization barriers.
Read how Graph Neural Networks in Action explains torch.distributed →
Related concepts
- DeviceMeshuses · A PyTorch distributed abstraction for arranging devices into a mesh and defining communication along its dimensions.
- distributed trainingused by · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
- model parallelismhas part · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
- MPIalternative to · The Message Passing Interface (MPI) is a standard specification for communication among processes in parallel and distributed-memory computing.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.