A failure occurs when a workflow component does not complete as intended. In machine-learning workflows, its impact depends on dependencies, available resources, persisted data, and downstream steps. Simple sequential pipelines can retry the failed step, but branching workflows may require decisions about which results remain usable and whether users can receive timely aggregated results. section 6.3
Handle a failed workflow step by following an operational diagnosis:
- Identify the root cause, such as a lost network connection, unavailable data, preempted workers, or insufficient memory. section 6.3.2
- Classify the failure as temporary or permanent before retrying; retries are ineffective when the underlying shortage or dependency loss persists. section 6.3.2
- Check whether required outputs still exist. Persisted data can allow training to restart, while data lost from memory requires data ingestion again. section 6.3.2
- Verify sufficient computational resources before rerunning training after worker preemption or an out-of-memory failure. section 6.3.2
- Record runtime metadata, including data availability and memory or CPU usage, so later failures can be interpreted rather than blindly retried. section 6.3.2
Parallel model-training workflow
Three training steps produce models with different tagging accuracies. A selection step chooses the best two models meeting a 90% threshold, two serving steps generate results, and an aggregation step presents them to users. If two training steps fail, rerunning everything wastes completed work; recovery instead depends on the causes, retained outputs, and remaining resources. section 6.3
