liveBook · Manning

Distributed programming

As explained in Graph Neural Networks in Action

Distributed processing is the execution of computation across multiple cooperating devices or processes.

Distributed programming executes computation across multiple cooperating devices or processes. For very large graphs, spreading training across processors or machines can reduce execution time when one system cannot handle the dataset efficiently. section 7.11 section 7.2.1

A practical distributed-training workflow typically follows these steps:

  • Distribute the dataset or training workload across multiple processors or cluster machines. section 7.3.1
  • Run forward propagation, compute loss, backpropagate, and update model parameters within the training loop. section 7.8.1
  • Synchronize processes at required points, such as before the next epoch, using PyTorch's distributed communication package and dist.barrier(). section 7.8.1
  • Start distributed execution behind an if __name__ == '__main__': guard and destroy the process group after training. section 7.8.1

Choosing a communication backend

Use NCCL for GPU-based distributed training and Gloo for CPU training. Both provide collective communication operations, while the machine or cluster must supply enough resources for parallel computation. section 7.8.1

Distributed execution improves speed but adds development, configuration, synchronization, and resource-management overhead. Sampling, batching, remote backends, sparse representations, and graph coarsening can be combined with it to address scale, memory, and fidelity trade-offs. section 7.11 section 7.3.1

How Graph Neural Networks in Action covers it

How deep it goes
Explains workload distribution, training-loop synchronization, entry-point guards, process-group cleanup, and NCCL or Gloo selection.
Examples
Uses PyTorch training loops, dist.barrier(), process groups, NCCL, Gloo, processor nodes, and machine clusters as implementation examples.
What it assumes
Assumes familiarity with graph training and computational resources while teaching synchronization and distributed-training setup explicitly.
What you can do after
Operate/configure distributed training with PyTorch communication primitives and choose NCCL or Gloo for GPU or CPU execution.
How it connects
Connects distributed computing with sampling, batching, remote backends, graph coarsening, sparse representations, and time and space complexity.

Also explained in

Compare how 3 books explain distributed programming →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Graph Neural Networks in Action. The section links point to the parts of the book it draws on.