Kubeflow training operators run framework-specific machine-learning training jobs on Kubernetes without requiring a custom training-service controller. Operators such as TensorFlow, PyTorch, MXNet, and MPI launch and monitor containerized training code for their respective frameworks. They fit teams that want a Kubernetes-native, cloud-agnostic training backend with distributed execution, scalability, and failure handling while avoiding the effort of building those controls from scratch. section 3.4 section 3.4.1 section 3.6
Each operator implements the Kubernetes controller pattern. A user declares a desired training job as a custom resource definition (CRD), such as TFJob, PyTorchJob, or MPIJob. The operator watches that resource, creates the required pods and services, supplies framework-appropriate environment variables and command-line arguments, monitors execution, and writes observed status back to the CRD. Reconciliation continually moves actual cluster resources toward the job specification. section 3.4.2 section 3.4.3
Figure 3.12 The Kubeflow training operator workflow. A user first creates a TFJob CRD object that defines a training request, and then the TensorFlow operator detects this object and creates actual pods to execute the TensorFlow training image. The TensorFlow operator also monitors the pod status and updates its status to the TFJob CRD object. The same workflow applies to the PyTorch operator.
A training job starts as a CRD, then the matching operator creates pods for the training image and updates job status as pod state changes. section 3.4.3
A practical operator workflow is:
- Install the desired standalone operator and its CRD in the Kubernetes cluster; installing the full Kubeflow system is not required. section 3.4.4
- Containerize training code and make runtime parameters available through environment variables or command-line arguments. section 3.4.1 section 3.4.4
- Create a framework job YAML resource defining replicas, restart policy, container configuration, environment, and training command; submit it with kubectl. section 3.4.4
- Query the job CRD for current status, which the controller continuously updates while managing queued work and replacing failed pods to restore the requested replica count. section 3.4.3 section 3.4.4
PyTorchJob as a training request
A PyTorchJob can declare one Master replica and one Worker replica, use OnFailure restart behavior, provide a credentials environment variable, and run a Python MNIST command with epoch and batch-size arguments. After kubectl creates the YAML resource, the PyTorch operator queues the request and allocates pods to execute it; kubectl can retrieve the updated job object as YAML. section 3.4.4
For an existing training platform, place a wrapper service in front of the operators. The wrapper translates the platform’s training requests into CRD CRUD operations, manages the job queue, and reads CRDs as the source of truth for training status. This preserves the existing front end while replacing or augmenting the computation backend with Kubernetes-managed training. section 3.4.5
