A parameter is a learned numerical value that determines part of a model’s behavior. In a neural-network layer, parameters appear as weights and biases; training updates them so modeled relationships better match the objective. section 3.2.2
Each training iteration uses input features to calculate a loss and gradients, then updates the parameters. Updating parameters across many layers can make training slow, motivating distributed designs. section 3.2.2
Distributed training stores and updates parameters in two practical ways:
- A parameter-server design keeps the model parameters on one or more dedicated servers while workers calculate gradients from separate data partitions and send updates. section 3.2.2
- A collective-communication design keeps a complete parameter copy on every worker, aggregates gradients with allreduce, and synchronizes identical updated models. section 3.8 section 7.3.2
- Parameter servers can become communication bottlenecks when many workers send gradients concurrently, whereas replicated parameters suit models that fit on each worker. section 7.3.2
LeNet and distributed classification
A LeNet model classifying YouTube videos learns layer weights and biases from video features. Workers process data partitions, calculate gradients, and update shared or replicated model parameters. section 3.2.2
Figure 7.10 Distributed model training component with only worker nodes, where every worker stores a copy of the entire set of model parameters and consumes partitions of data to calculate the gradients
Shows workers retaining complete copies of the parameters while consuming data partitions to calculate gradients. section 7.3.2
