Priority-based scheduling orders queued work by assigned priority rather than first-come, first-served order. When capacity is scarce, a higher-priority job can run first; it may also preempt or evict lower-priority work already using the required compute resources. This provides an escape from workloads that are blocked by strictly sequential scheduling and helps keep urgent work moving. section 6.2.2 section 6.7
In a shared machine-learning cluster, priority is an operational control, not a guarantee of fairness. Unrestricted top-priority submissions can cause other jobs to be evicted or never scheduled, so administrators need rules and limits on high-priority job creation. section 6.2.3
Use priority-based scheduling when the system needs to:
- Run urgent jobs ahead of lower-importance queued work when resources are released. section 6.2.2
- Reclaim resources from a running low-priority job for newly submitted high-priority work. section 6.2.2
- Complement fair-share scheduling where first-come, first-served execution can leave no job able to proceed. section 6.2.2
Ordering and preemption
For four concurrent jobs, A and C marked high priority run before D, marked medium, and B, marked low. If low-priority B is already running when high-priority A arrives, B can be preempted: its resources are released and assigned to A. section 6.2.2
Figure 6.8 The order of execution for the four jobs (A, B, C, and D) where the running low-priority job is preempted by a new job with higher priority
This execution order illustrates a newly arrived high-priority job preempting a running low-priority job when compute resources are unavailable. section 6.2.2
