Gang scheduling

As explained in Distributed Machine Learning Patterns

Gang scheduling starts or schedules a group of cooperating workers together so they are simultaneously available for coordinated computation or communication.

Gang scheduling starts communicating workers only when the full required group is available and ready. It fits distributed model training in which workers must exchange data together; starting only part of the group leaves allocated workers idle while the task waits. section 6.2.2

The pattern is especially useful for collective communication such as allreduce. Each worker must exchange calculated gradients and update its model; if network instability leaves some workers unready, dependent processes can miss gradient values and the collective operation remains blocked. section 6.2.2

Allreduce after network recovery

A process group has workers performing allreduce while an unstable network prevents two workers from becoming ready. Rather than start ready workers and let them wait, gang scheduling holds every worker until the network stabilizes and the whole group can send gradients together. section 6.2.2

Figure 6.10 With gang scheduling, the worker processes will not start sending the gradients until they are all in the ready state after the network becomes stable.

This diagram shows the gating behavior: no worker sends gradients until every process reaches the ready state after the network becomes stable. section 6.2.2

Choose gang scheduling when coordinated communication outweighs the cost of waiting for the complete group:

• Use it for collective communication-based training, where all workers must participate to exchange gradients without wasting ready workers' compute resources. section 6.2.3
• Do not use it when elastic scheduling can begin useful training with whatever number of workers is currently available. section 6.2.3
• Account for training-strategy changes under elastic scheduling, since varying worker counts can change batch size and affect training accuracy. section 6.2.3

Gang scheduling is one scheduling option alongside fair-share and priority scheduling. These patterns address operational concerns in machine-learning systems, including resource starvation and deadlocks, but gang scheduling specifically coordinates workers that depend on one another for communication. section 6.0 section 6.7

How Distributed Machine Learning Patterns covers it

How deep it goes

Explains readiness gating through an unstable-network allreduce scenario and contrasts it with elastic scheduling.

Examples

Illustrates a worker process group whose unstable network blocks gradient exchange until every worker becomes ready.

What it assumes

Teaches coordinated readiness using distributed training workers and allreduce without requiring stated prior scheduling knowledge.

What you can do after

Evaluate/understand it by deciding whether coordinated worker readiness or elastic scheduling better serves a distributed training job.

How it connects

Connects collective communication and allreduce with elastic scheduling, batch size, learning-rate scheduling, and priority scheduling.

liveBookmanning.com

Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

Related concepts

• workeruses · A worker is a person or computational execution unit assigned to perform a task as part of a larger system or activity.
• deadlockaddresses · A deadlock is a state in concurrent execution where two or more activities are indefinitely blocked by dependencies on resources, events, or progress that cannot become available.
• distributed traininguses · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.