A memory store temporarily holds training-job objects and their execution-state collections. It separates waiting, launching, running, and finalized jobs so the training service can coordinate requests, execution, and status queries. section 3.3.2
The reference implementation uses four maps to represent the job lifecycle:
- The job queue holds submitted jobs waiting for available Docker capacity. section 3.3.2 section 3.3.4
- The launching list holds jobs whose Docker containers are being started. section 3.3.2 section 3.3.4
- The running list holds jobs whose containers are executing. section 3.3.2 section 3.3.4
- The finalized list holds completed jobs, whether training succeeded or failed. section 3.3.4 section 3.3.5
The Docker job tracker monitors the queue, launches containers when capacity exists, and moves each job between maps as Docker reports state changes. A status request identifies the map containing the job ID and derives the current training status from it. section 3.3.4 section 3.3.5
Tracking a submitted training job
A Train API request receives a job ID when the request enters the job queue. After capacity becomes available, the tracker launches its Docker container and moves the job to launching, then running. Once the container finishes, the job enters finalized; a later status query reports waiting, launch, running, success, or failure according to its map. section 3.3.4 section 3.3.5
Figure 3.5 A high-level service design and user workflow: user training requests are queued, and the Docker job tracker picks up jobs from the queue and launches Docker containers to run model training.
Use the service workflow to trace how queued training requests pass through the memory store and Docker job tracker. section 3.3.2
