Collective communication coordinates distributed training without parameter servers when a model can fit on each worker. Every worker keeps a full model copy, trains on a data partition, and exchanges gradients with the worker group so every copy receives the same update. section 3.3.2 section 7.3.2
It addresses contention created when several workers push gradients to the same parameter server. Rather than tuning a worker-to-server ratio or routing updates through model partitions, workers aggregate their gradients together and update identical local parameter sets. section 3.3 section 3.3.3
The synchronization flow uses collective operations across a process group:
- Reduce gradients with an operation such as sum, maximum, minimum, or average. section 3.3.2
- Broadcast the reduced gradients to every worker so their model parameters are updated consistently. section 3.3.2
- Use allreduce to combine reduction and distribution of results to all processes. section 3.3.2
Medium-size Fashion-MNIST model
A 30 MB compressed classification model can use worker-only training: each worker computes gradients from its data share, then allreduce communicates aggregated gradients before every worker updates its complete model copy. section 7.3.2
The pattern removes communication overhead between workers and parameter servers and makes adding workers simpler for small or medium-sized models. Its trade-off is peer communication: ordinary allreduce can slow as worker count rises because each worker communicates with all peers. Ring-allreduce reduces peer connections to two and is bandwidth-optimal for sufficiently large aggregated gradients. section 3.3.3 section 3.9
Figure 3.20 An example of an allreduce operation that reduces the results on each process in the group and then sends the result to every process in the group
View allreduce as one operation that aggregates values from every process and returns the result to every process. section 3.3.2
