Broadcast operation

As explained in Distributed Machine Learning Patterns

A collective communication operation that sends one value or result to every process in a participating process group.

A broadcast operation sends one value or result to every process in a participating process group. In distributed training, workers receive reduced gradients so they can update identical, current model copies. section 3.3.2

A broadcast operation participates in a distributed training workflow:

• Workers calculate gradients from their data partitions. section 3.3.2
• A reduce operation combines gradients using a function such as sum, maximum, minimum, or average. section 3.3.2
• The broadcast operation sends the reduced result to every worker, completing an allreduce cycle. section 3.3.2
• Workers begin the next round with the updated model, without requiring parameter servers. section 3.3.2

Synchronous worker update

Three workers calculate gradient vectors. A reduce operation merges them through the first process, and a broadcast operation distributes the reduced gradients to all workers so each stores the same model. section 3.3.2 section 3.7

Figure 3.19 An example of a broadcast operation that sends a value to every process in the process group

Use the figure to trace one value moving from a process to every member of the process group. section 3.3.2

How Distributed Machine Learning Patterns covers it

How deep it goes

Explains the operation within an allreduce workflow and stops short of implementation-specific communication details.

Examples

Illustrates process-group distribution, gradient vectors, repeated worker updates, and an allreduce sequence.

What it assumes

Assumes workers, process groups, gradients, and reduce operations as the surrounding distributed-training vocabulary.

What you can do after

Apply it to distribute aggregated gradients and keep replicated worker models identical.

How it connects

Connects broadcast operation to reduce operation, allreduce, collective communication, distributed training, and parameter servers.

liveBookmanning.com

Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

Related concepts

• allreducepart of · Allreduce is a collective communication operation that combines corresponding values across participating processes or devices and distributes the resulting aggregate to every participant.

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.