Process group

As explained in Distributed Machine Learning Patterns

A worker process group is a set of worker processes that participate together in distributed computation and collective communication.

A process group is a set of worker processes that participate together in collective communication during distributed model training. Each worker computes gradients, and the group coordinates their exchange so workers maintain the same model parameters. section 3.3.2

A process group supports distributed training through a repeated collective workflow:

• Workers consume data partitions and calculate gradients independently. section 3.3.2
• A reduce operation combines corresponding gradients using functions such as sum, maximum, minimum, or average. section 3.3.2
• A broadcast operation distributes the reduced result to every worker. section 3.3.2
• Together, reduce and broadcast form allreduce, allowing every worker to update an identical model before the next round. section 3.3.2

Recovering from a stalled allreduce

Three workers enter an allreduce operation, but an unstable network slows two communications and blocks the group. Abandoning the affected workers and the stalled operation allows the remaining workers to form a new process group and continue training from the existing model. The same reconstruction approach handles worker preemption without restarting training from scratch. section 3.4.2

Figure 3.20 An example of an allreduce operation that reduces the results on each process in the group and then sends the result to every process in the group

Use the allreduce diagram to trace how each process reduces results and receives the combined value. section 3.3.2

How Distributed Machine Learning Patterns covers it

How deep it goes

Explains reduce, broadcast, allreduce, synchronization, and reconstruction after failures, but stops short of implementation configuration.

Examples

Uses three-worker allreduce, unstable-network delays, stalled communication, worker preemption, and reconstructed groups as operational examples.

What it assumes

Assumes distributed workers and gradients, while explaining collective communication through reduce, broadcast, and allreduce.

What you can do after

Diagnose/debug with it by identifying blocked collective communication and rebuilding the worker group around surviving processes.

How it connects

Connects distributed training with collective communication, gradients, reduce, broadcast, allreduce, and model parameters.

liveBookmanning.com

Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Elixir in Action, Third EditionA process group is a named cluster-wide collection of processes that can be queried as a set for communication or coordination.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionThe set of processes participating in a coordinated distributed-training run and sharing communication context.

Compare how 3 books explain process group →

Related concepts

• :pgimplemented by · :pg is an Erlang library module for creating cluster-wide process groups and managing their membership.
• allreduceuses · Allreduce is a collective communication operation that combines corresponding values across participating processes or devices and distributes the resulting aggregate to every participant.
• DeviceMeshimplemented by · A PyTorch distributed abstraction for arranging devices into a mesh and defining communication along its dimensions.
• torch.distributed.init_process_groupimplements · A PyTorch distributed-training function that initializes the process group and establishes communication among participating processes.
• distributed trainingused by · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
• process registrykind of · A process registry is a component that associates application-defined names or keys with process identifiers for registration and lookup.

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.