World size

As explained in Deep Learning with PyTorch, Second Edition

The total number of participating processes in a distributed training job.

World size is the total number of processes participating in a distributed training job. When one GPU is assigned to each process, it generally also equals the number of GPUs in use. section 16.1.1

Figure 16.1 Single GPU training (left) vs. multi-GPU training (center) vs. training with multiple machines and multiple GPUs (right)

Compare how world size changes from one GPU to several GPUs on one machine and then to several machines. section 16.1.1

The examples distinguish common distributed-training setups:

• Single-GPU training uses a world size of 1. section 16.1.1
• Training on four GPUs in one machine uses a world size of 4. section 16.1.1
• Training on two machines with four GPUs each uses a world size of 8. section 16.1.1
• Each process also receives a unique rank, ranging from 0 through world size minus 1. section 16.1.1

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes

Clarifies process counting and GPU correspondence, but stops short of detailing communication or process-launch mechanics.

Examples

Uses world sizes of 1, 4, and 8 for single-GPU, single-machine, and multimachine training.

What you can do after

Operate/configure distributed training by identifying the participating process count and relating it to GPU layout.

How it connects

Connects world size with rank, process groups, nodes, data parallelism, and collective communication.

liveBookmanning.com

Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Related concepts

• distributed trainingconfigures · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
• rankcontrasts with · A rank is a unique identifier assigned to a participating process or worker within a distributed computation or communication group.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.