A custom resource definition (CRD) extends the Kubernetes API with a custom resource kind and schema. A CRD object records desired state; a controller watches it, creates or changes actual Kubernetes resources, and updates observed status. section 3.4.2
A CRD participates in an operator architecture with these responsibilities:
- The user submits a custom resource object describing a request, such as a training job or workflow. section 3.4.2 section 3.4.3
- A controller or operator watches that object and provisions the Kubernetes services, pods, containers, or volumes required to realize it. section 3.4.2 section 3.4.3
- The controller continuously reconciles desired and observed state, replacing failed pods and publishing current status through the custom resource. section 3.4.2 section 3.4.3
Training and workflow resources
Kubeflow training operators watch framework-specific CRD objects such as TFJob, PyTorchJob, and MPIJob, then create training pods with the required commands and environment variables. Argo Workflows likewise represents workflows and tasks as CRD objects, executing each task in a Kubernetes pod. section 3.4.3 section 9.3.2
Figure 3.12 The Kubeflow training operator workflow. A user first creates a TFJob CRD object that defines a training request, and then the TensorFlow operator detects this object and creates actual pods to execute the TensorFlow training image. The TensorFlow operator also monitors the pod status and updates its status to the TFJob CRD object. The same workflow applies to the PyTorch operator.
Trace the CRD workflow from a submitted TFJob object through operator reconciliation, pod creation, monitoring, and status updates. section 3.4.3
