Distributed programming executes computation across multiple cooperating devices or processes. For very large graphs, spreading training across processors or machines can reduce execution time when one system cannot handle the dataset efficiently. section 7.11 section 7.2.1
A practical distributed-training workflow typically follows these steps:
- Distribute the dataset or training workload across multiple processors or cluster machines. section 7.3.1
- Run forward propagation, compute loss, backpropagate, and update model parameters within the training loop. section 7.8.1
- Synchronize processes at required points, such as before the next epoch, using PyTorch's distributed communication package and dist.barrier(). section 7.8.1
- Start distributed execution behind an if __name__ == '__main__': guard and destroy the process group after training. section 7.8.1
Choosing a communication backend
Use NCCL for GPU-based distributed training and Gloo for CPU training. Both provide collective communication operations, while the machine or cluster must supply enough resources for parallel computation. section 7.8.1
Distributed execution improves speed but adds development, configuration, synchronization, and resource-management overhead. Sampling, batching, remote backends, sparse representations, and graph coarsening can be combined with it to address scale, memory, and fidelity trade-offs. section 7.11 section 7.3.1


