Distributed training

As explained in Deep Learning with PyTorch, Second Edition

A training approach that distributes computation or data across devices or machines.

Distributed training spreads model computation or data across multiple GPUs or machines. It becomes useful when models or datasets exceed what one GPU can handle, and can improve training speed and hardware utilization. section 16.0 section 16.12

A distributed system contains processes identified by rank, with world size counting all processes. Processes communicate through a process group; a node is one machine, and local rank identifies a process within that machine. section 16.1.1

Common parallelism strategies divide different parts of training:

• Data parallelism replicates the model on each device, splits the data, and synchronizes gradients. section 16.12
• Model parallelism divides model parameters and computation across devices, while pipeline parallelism divides the model into sequential stages and processes concurrent microbatches. section 16.12
• Tensor parallelism partitions layers or tensors; fully sharded data parallelism shards model and data; n-dimensional parallelism combines several strategies. section 16.12

Figure 16.1 Single GPU training (left) vs. multi-GPU training (center) vs. training with multiple machines and multiple GPUs (right)

Compare single-GPU training with multi-GPU training on one machine and multi-machine training with multiple GPUs. section 16.1.1

A practical distributed job

A PyTorch training script initializes a process group, launches multiple processes—often with torchrun—and uses collective operations such as broadcast and allreduce to synchronize data and gradients. TorchTitan provides a minimal native-PyTorch example that can launch a large-language-model training job with one command. section 16.12 section 16.9

Multiple GPUs are typical, but the examples can also run on CPUs for experimentation. CPU runs may not provide meaningful speedups. Linux or macOS is recommended because PyTorch distributed APIs have limited Windows support. section 16.1.2

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes

Explains terminology, process initialization, communication, and major parallelism strategies, then stops short of exhaustive engineering detail.

Examples

Uses single-, multi-GPU, and multi-machine layouts, CPU experiments, torchrun jobs, and the TorchTitan training repository.

What it assumes

Introduces distributed terminology and process isolation while assuming familiarity with model training and basic Python execution.

What you can do after

Operate/configure PyTorch jobs with process groups, multiple processes, collective communication, and selected parallelism strategies.

How it connects

Connects process groups and collective communication with data, model, pipeline, tensor, and fully sharded data parallelism.

liveBookmanning.com

Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Distributed Machine Learning PatternsThe process of fitting a machine learning model by coordinating computation across multiple machines or execution units.
• 

  liveBookmanning.com
  Designing Deep Learning Systems: A guide for software engineersDistributed training is the process of training a machine learning model across multiple computing devices or processes, often with data partitioned among workers.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA training approach that spreads computation and/or data across multiple devices to speed up model training.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionDistributed training is the execution of model training across multiple processors or machines that coordinate computation and parameter updates.
• 

  liveBookmanning.com
  Graph Neural Networks in ActionDistributed training is a technique in which model computation and data processing are divided across multiple computing processes or devices.

Compare how 6 books explain distributed training →

Related concepts

• data parallelincludes · Data parallelism is a parallel-computing technique that applies the same computation to separate portions of data concurrently, often by replicating the computation across workers.
• model parallelismincludes · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
• pipeline parallelismincludes · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
• TF_CONFIGconfigured by · TF_CONFIG is an environment-variable configuration that specifies cluster and task information for TensorFlow distributed execution.
• distribution strategykind of · A distribution strategy is a mechanism that specifies how computation and model variables are replicated and coordinated across devices or workers.
• model training serviceused by · A service that provides computation resources and job management for executing model-training code, producing trained models, and tracking training execution.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.