A stale gradient is calculated from a model version that is no longer current when a worker sends it to a parameter server. Concurrent workers can therefore propose updates based on different model states. section 3.3.2
The impact depends on how far model versions have diverged and how parameters are partitioned. A stale gradient affecting a large or unevenly stored partition can substantially change training performance. section 3.3.2
Handle stale gradients by weighing statistical damage against recomputation cost:
- Drop the gradient when its outdated model version could strongly distort the update, then let another worker provide a current gradient. section 3.3.2
- Retain the gradient when recomputing it would waste substantial work, such as when it represents a large portion of the training data. section 3.3.2
Figure 3.14 The second worker is trying to push gradients calculated from half of the training data.
Shows a worker pushing gradients calculated from half of the training data, making the recomputation-versus-retention trade-off concrete. section 3.3.2
