A deadlock occurs when jobs cannot proceed because required resources or progress remain unavailable. In machine-learning workflows, scheduling choices can create this state even when recovery work is waiting in the queue. section 6.0 section 6.2.2
One failure pattern begins when a data-producing job becomes stuck on an unstable database connection. A maintenance job is needed to kill and restart it, but fair-share scheduling queues that job behind other resource users, leaving no job able to advance. section 6.2.2
Scheduling strategies reduce this risk in different ways:
- Priority scheduling runs higher-priority jobs first and can preempt or evict running jobs to free computational resources. section 6.2.2
- Gang scheduling coordinates distributed workers so they begin a collective operation only when all workers are ready, avoiding partial progress during network instability. section 6.0
- Fair-share scheduling supports resource allocation but can contribute to deadlock when first-come, first-served ordering delays essential recovery work. section 6.2.2
Queued recovery job
Job 2 repeatedly reconnects to an unstable database and prevents job 1 from receiving data. An administrator submits job 3 to restart job 2, but insufficient resources and queue ordering leave the restart itself waiting, so the workflow remains blocked. section 6.2.2
