A worker is an execution unit that takes assigned data or work and performs computation independently. Workers make distributed systems faster by processing partitions in parallel rather than moving one model or task sequentially from machine to machine. section 1.1.2 section 1.2.2
In distributed training, a worker node consumes a subset of the dataset, calculates gradients, and sends them for model updates. With a parameter server, workers offload parameter storage and updates to dedicated servers; with collective communication, each worker instead keeps a full model copy. section 3.2.2 section 7.3.2
Adding workers improves throughput, but introduces coordination decisions:
- Workers may finish their partitions at different times, so completion order can affect both resource use and which gradients are applied first. section 1.1.2
- Asynchronous workers can compute without waiting on one another, but may use model partitions that are no longer current. section 3.2.2 section 7.3.2
- Too many workers sending gradients to the same parameter server can create blocking communication and a bottleneck. section 7.3.2
- For independent batch jobs, a work queue keeps workers busy by distributing separate work items and allows worker capacity to scale up or down. section 1.2.2
Gradient workers with a parameter server
For a LeNet training workload that fits on existing resources, one server stores model parameters while multiple workers split the dataset. Each worker calculates gradients for its partition and sends results to the server; ignoring message-passing costs, adding workers can yield roughly triple speedup. section 3.2.2
Figure 7.10 Distributed model training component with only worker nodes, where every worker stores a copy of the entire set of model parameters and consumes partitions of data to calculate the gradients
Compare worker-only distributed training: every worker stores the full parameter set, consumes a data partition, and calculates gradients. section 7.3.2
