A worker group is a set of parallel workers that provides the compute capacity for a distributed training run or workflow task. Its size can be selected for a training request, while the service manages allocation and communication across workers. section 4.3.1 section 9.2.2
A worker group fits into the surrounding system as follows:
- Runs training jobs through a Docker engine or Kubernetes cluster; the Kubernetes job tracker launches pods and monitors execution status. section 4.3.1
- Abstracts the underlying infrastructure from workflow tasks, allowing Kubernetes workers and Amazon EC2 workers to execute the same task. section 9.2.2
- Reads predecessor outputs and saves task results through shared object storage, while centralized metadata storage decouples workers from other orchestration components. section 9.2.2 section 9.2.2
Distributed training request
Alex submits the same training-service API request used for single-device training, but sets PARALLEL_INSTANCES to define the worker-group size. The service then runs the distributed training job through its selected backend. section 4.3.1
Figure 4.3 (a) The previous training service design introduced in figure 3.5; (b) the updated service design with distributed training support in Kubernetes
Use the architecture diagram to trace how the worker group and Kubernetes job tracker extend a training service that also supports Docker execution. section 4.3.1
