Fault tolerance keeps distributed machine-learning workloads operating or recoverable when workers, nodes, or network operations fail. Without it, one failed worker can waste the work completed by every other worker and reduce service availability. section 4.2.2 section 5.3.1
Implement fault tolerance by preserving and restoring execution state:
- Save each worker’s training state, including model parameters, to a remote filesystem after training steps. section 4.2.2
- Restart a failed or stalled worker and restore its most recently saved state. section 4.2.2
- Provide backup-storage configuration to the training container and use framework support for checkpoint backup and restoration. section 4.2.2
- Record the latest state of each hyperparameter-optimization trial so failed sequential experiments can resume automatically. section 5.3.1
Sequential HPO recovery
An HPO algorithm may require trial 2 to follow trial 1 because it uses earlier hyperparameters and results. If a node restart or network problem interrupts trial 1, restoring its latest checkpoint prevents the entire experiment from failing. section 5.3.1
