Process group
A process group is a defined collection of processes that share a communication or coordination context for distributed computation.
Distributed Machine Learning Patterns
A worker process group is a set of worker processes that participate together in distributed computation and collective communication.
- How it is used
- Coordinates synchronous gradient exchange among distributed workers that train the same machine-learning model.
- How deep it goes
- Explains reduce, broadcast, allreduce, synchronization, and reconstruction after failures, but stops short of implementation configuration.
- Examples
- Uses three-worker allreduce, unstable-network delays, stalled communication, worker preemption, and reconstructed groups as operational examples.
- What you can do after
- Diagnose/debug with it by identifying blocked collective communication and rebuilding the worker group around surviving processes.
Read how Distributed Machine Learning Patterns explains process group →
Elixir in Action, Third Edition
A process group is a named cluster-wide collection of processes that can be queried as a set for communication or coordination.
- How it is used
- Coordinates replicated processes across a cluster for discovery, broadcasting updates, and surviving node failures.
- How deep it goes
- Explains startup, joining, membership queries, and update coordination, but stops short of detailed failure semantics.
- Examples
- Uses two connected nodes, Bob's replicated to-do list, :pg.join/2, and :pg.get_members/1.
- What you can do after
- Apply it via :pg to register distributed replicas, inspect group membership, and broadcast requests across processes.
Read how Elixir in Action, Third Edition explains process group →
Deep Learning with PyTorch, Second Edition
The set of processes participating in a coordinated distributed-training run and sharing communication context.
- How it is used
- Coordinates tensor communication among processes participating in distributed model training.
- How deep it goes
- Explains initialization, backend selection, and key collective operations without descending into lower-level communication implementation.
- Examples
- Uses PyTorch broadcast and allreduce code plus diagrams comparing multi-GPU and multi-machine process layouts.
- What you can do after
- Operate/configure distributed training by initializing a process group and applying collective communication operations.
Read how Deep Learning with PyTorch, Second Edition explains process group →
Related concepts
- :pgimplemented by · :pg is an Erlang library module for creating cluster-wide process groups and managing their membership.
- allreduceuses · Allreduce is a collective communication operation that combines corresponding values across participating processes or devices and distributes the resulting aggregate to every participant.
- DeviceMeshimplemented by · A PyTorch distributed abstraction for arranging devices into a mesh and defining communication along its dimensions.
- torch.distributed.init_process_groupimplements · A PyTorch distributed-training function that initializes the process group and establishes communication among participating processes.
- distributed trainingused by · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
- process registrykind of · A process registry is a component that associates application-defined names or keys with process identifiers for registration and lookup.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.