Training state is the saved model information associated with a training iteration. In distributed training, preserving each worker’s latest state lets a failed worker restart without discarding the other workers’ computation. section 4.2.2 section 4.2.4
A practical training-state design assigns responsibilities across the training service and training code:
- Provide remote storage, such as a remote filesystem, for each worker’s saved state. section 4.2.2
- Pass storage access configuration into the training container. section 4.2.2
- Save the training state after each iteration through a backup API or framework feature. section 4.2.4
- Restore the most recent previous state when a worker fails or exceeds the iteration deadline, then restart that worker. section 4.2.2 section 4.2.4
Recovering a failed worker
A distributed training group saves each worker’s neural-network state after every iteration. If one worker fails during the next iteration, the service restores its previous state and lets the group continue instead of wasting completed computation. section 4.2.4
