A distributed training group is the registered set of training processes that coordinate one data-parallel run. Each process joins with a unique rank; the group’s world size identifies the total process count. Registration waits until all workers join, allowing processes on one machine or multiple machines to communicate as one run. section 4.2.3 section 4.3.5
The group is the boundary that lets a framework synchronize work. In PyTorch, initializing the process group precedes wrapping a model in DistributedDataParallel (DDP), which handles communication, gradient aggregation, and local parameter updates. A distributed sampler gives each process an exclusive dataset subset. section 4.2.3 section 4.3.5
A training service can hide group setup from training-code users by doing the following:
- Provision the requested number of worker resources and assign their addresses and identities. section 4.2.4
- Set framework-specific configuration, such as PyTorch master address, port, rank, and world size, or TensorFlow TF_CONFIG. section 4.2.3
- Start the workers, then let the framework coordinate synchronous data-parallel execution. section 4.2.4
- Assign rank 0 as master when a single worker must perform a shared final action, such as uploading the final model. section 4.3.5
Three-worker Kubernetes run
A request with PARALLEL_INSTANCES=3 creates a three-worker group rather than a single-device run. Kubernetes exposes two worker pods, one master pod, and a service for master-to-worker network connectivity; training status remains available through the service API. section 4.3.2
def setup(rank, world_size):
os.environ['MASTER_ADDR'] = 'xxxx'
os.environ['MASTER_PORT'] = 'xxx'
# initialize the process group, "gloo" is one of the communication
# backends Pytorch supports, it also supports MPI and NCCL.
# rank is the process’s rank, it's a globally unique id
# for this process. rank=0 means master process.
# world_size is the total number of processes in this training group.
dist.init_process_group("gloo", rank=rank, world_size=world_size)
def cleanup():
dist.destroy_process_group()
This initialization pattern shows the minimum PyTorch group identity: a communication backend, global rank, world size, and master endpoint. It also includes explicit teardown after the run. section 4.2.3
