Distributed training

As explained in Graph Neural Networks in Action

Distributed training is a technique in which model computation and data processing are divided across multiple computing processes or devices.

Distributed training divides model computation and data processing across multiple processes or devices. In PyTorch DistributedDataParallel (DDP), each GPU works on different data while synchronized gradients improve a shared model, making training faster than single-GPU or simpler parallel approaches. section 7.8.1

A practical DDP training workflow is:

• Instantiate the model, then wrap it with PyTorch DistributedDataParallel. section 7.8.1
• Initialize distributed communication and assign each process its device and rank. section 7.8.2
• Load data, run forward and backward passes, compute loss, and update parameters on each process’s data subset. section 7.8.1 section 7.8.2
• Synchronize processes at required points, such as a dist.barrier() before the next epoch, then destroy the process group after training. section 7.8.1
• Choose NCCL for GPU collective communication or Gloo for CPU training, and provide sufficient machine or cluster resources. section 7.8.1

Two-process MNIST training

A PyTorch example initializes a DDP environment with NCCL, loads MNIST through DataLoader, assigns each spawned process a CUDA device, wraps a Flatten-plus-Linear network in DistributedDataParallel, and trains across two GPUs. A related GCN example uses the Amazon Products dataset and NeighborLoader on a dual-GPU Kaggle notebook. section 7.8.2

Figure 7.3 Process diagram for initiating and running a training with multiple processor devices

Use the process diagram to follow initialization and execution across multiple processor devices. section 7.8.2

How Graph Neural Networks in Action covers it

How deep it goes

Explains DDP setup, data partitioning, gradient synchronization, process barriers, multiprocessing, and communication backends.

Examples

Uses a two-GPU MNIST network and a related GCN example with Amazon Products and NeighborLoader.

What it assumes

Assumes access to suitable multi-device resources while establishing model, data-loader, process-group, and entry-point setup explicitly.

What you can do after

Operate/configure PyTorch DDP training with spawned processes, synchronized gradients, device assignment, barriers, and backend selection.

How it connects

Connects DistributedDataParallel with gradient synchronization, process groups, DataLoader, multiprocessing, NCCL, Gloo, and NeighborLoader.

liveBookmanning.com

Read Graph Neural Networks in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Distributed Machine Learning PatternsThe process of fitting a machine learning model by coordinating computation across multiple machines or execution units.
• 

  liveBookmanning.com
  Designing Deep Learning Systems: A guide for software engineersDistributed training is the process of training a machine learning model across multiple computing devices or processes, often with data partitioned among workers.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA training approach that spreads computation and/or data across multiple devices to speed up model training.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionDistributed training is the execution of model training across multiple processors or machines that coordinate computation and parameter updates.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA training approach that distributes computation or data across devices or machines.

Compare how 6 books explain distributed training →

Related concepts

• data parallelincludes · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
• model parallelismincludes · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
• pipeline parallelismincludes · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
• trainingkind of · Training is a process of developing or improving a person’s or model’s capabilities through instruction, experience, or adjustment using data.
• TF_CONFIGconfigured by · TF_CONFIG is an environment-variable configuration that specifies cluster and task information for TensorFlow distributed execution.
• Kubeflowused by · Kubeflow is an open-source platform for developing, deploying, and orchestrating machine-learning workflows on Kubernetes.

This explainer was generated by Manning's knowledge graph from the text of Graph Neural Networks in Action. The section links point to the parts of the book it draws on.