A Kubernetes job tracker launches training workloads as Kubernetes pods, monitors their execution, and updates job state in an in-memory store. It lets one training API support both Docker and Kubernetes backends while handling single-device and distributed runs. section 4.3.1
A tracker coordinates the training workflow by:
- Watching the job queue and starting waiting jobs when cluster capacity is available. section 4.3.3
- Creating one pod for single-device training or multiple pods for distributed training, with one training process per pod. section 4.3.3
- Creating a Kubernetes Service for the master pod and passing WORLD_SIZE, RANK, MASTER_ADDR, and MASTER_PORT to every pod. section 4.3.3
- Moving jobs among launching, running, and finalized queues as pod execution changes. section 4.3.4
Distributed pod launch
For a three-process run, the tracker creates a master Service and assigns ranks 0, 1, and 2. Each pod receives WORLD_SIZE=3, the master Service DNS name, and a shared master port. section 4.3.3
Figure 4.5 Track the Kubernetes training job status: step 1, obtains the jobs in the running list; step 2, queries the pod execution status of each of the jobs running in the Kubernetes cluster; and step 3, moves the job to the finalized job list if the pod execution is complete (success or failure).
Use the status-tracking workflow to follow polling, master-pod representation of synchronous workers, and finalization after success or failure. section 4.3.4
