A job queue holds submitted training requests until execution capacity is available. In a training service, it is part of an in-memory job-tracking store that separates waiting work from launching, running, and finalized work; a database is the preferred production replacement for the in-memory store. section 3.3.2
Figure 3.6 The training job launching workflow: the Docker job tracker launches training containers from the job queue when it has the capacity.
The queue decouples request submission from container execution: the tracker watches waiting jobs and launches one when Docker capacity is available. section 3.3.4
A queue entry moves through explicit execution states:
- Waiting — the Train API stores request metadata and assigns a job ID for later status lookup. section 3.3.3 section 3.3.4
- Launching — after the tracker starts a Docker container, it moves the job from the queue to a launching list. section 3.3.4
- Running or finalized — container-status polling moves jobs to the running list, then to finalized success or failure. section 3.3.4
Submit, launch, and query a training job
A caller submits algorithm, dataset version, and hyperparameters through Train and receives a job_id. While the ID remains in the job queue, GetTrainingStatus reports waiting; list membership then reports launch, running, success, or failure as the tracker updates the job. section 3.3.3 section 3.3.5
Queue design also affects multi-tenant resource isolation. A single heavy user can fill a shared waiting queue and delay others. Dedicated machine pools with dedicated queues isolate teams, though fixed pools can leave capacity idle; Kubernetes namespaces are identified as the ideal approach for virtual clusters. section 3.3.7
For Kubernetes-based training, a wrapper service can manage the queue while translating incoming requests into Kubeflow training-operator CRD objects and reading status back from those objects. This avoids building a production training backend from scratch. section 3.4.5
