Kubeflow is an open-source ML platform on Kubernetes. Its Kubeflow Pipelines component, or KFP, orchestrates repeatable ML workflows: instead of manually coordinating data preparation, feature retrieval, inference, and output storage, define each task as a component and connect the components into a pipeline. Kubeflow also provides notebooks; a feature store is an example of a capability to integrate separately rather than expect from the platform. section 1.3 section 5.0
Build a KFP workflow in layers:
- Package each task as a reusable component. A component runs in its own Kubernetes pod. One approach puts Python code in a container image and declares its command, inputs, and outputs in component YAML; another uses a Python function decorated with @dsl.component and specifies its base image and dependencies. section 5.1.1 section 7.3.1
- Pass datasets through declared outputs and inputs, rather than relying on one task’s local files being available to the next. Keep components independent and generic so they can be tested, maintained, and reused. section 5.1.1 section 5.2
- Define the workflow with @dsl.pipeline, load or call its components, and connect a task’s outputs to downstream inputs. These connections form a directed acyclic graph: a dependent task waits for its predecessors. Expose values that must change between runs as pipeline parameters. section 5.1.2
- Compile the pipeline to YAML, then upload and run it in the Kubeflow UI or use the SDK. The UI provides a pipeline view, run status, and component logs. section 5.1.2 section 7.4
Batch income classification
An inference pipeline reads user IDs from MinIO, retrieves their features from Feast, obtains a model through MLflow, runs inference, and writes predictions to MinIO. The stages become four connected components, with intermediate datasets passed to subsequent steps. A drift-detection component can be inserted between feature retrieval and inference; if it detects drift against reference data, pipeline execution stops. section 5.1.1 section 6.2.2
Figure 5.2 Kubeflow pipeline interactions with MinIO, Feast, and MLflow. All the intermediate datasets are stored in MinIO to be retrieved by subsequent steps. The model and features are retrieved from MinIO and Feast, respectively.
Trace the batch pipeline’s connections to MinIO, Feast, and MLflow, including where intermediate datasets are stored for downstream steps. section 5.1.1
Kubeflow notebooks offer a Jupyter-like workspace on the same cluster as pipelines. When launching one, check its namespace, image, resource limits, and accelerator allocation: notebook pods can occupy resources needed elsewhere. Treat example credentials as development conveniences, not production configuration; use a secret-injection approach instead. section 7.1.1 section 7.3.1
