A pipeline component is a reusable unit of work that performs one ML task. Kubeflow Pipelines (KFP) connects components to turn manual steps—such as reading data, retrieving features, running inference, and writing predictions—into an executable workflow. Each component runs in its own Kubernetes pod, so its inputs, outputs, and dependencies must be explicit. section 5.1.1 section 5.1
Build a component around a clear task and data contract:
- Implement the task in Python. For a data-reading step, download a file from MinIO and write the resulting DataFrame to the designated output path so a later step can use it. section 5.1.1
- Package its runtime. A container-based component uses an image containing its code and dependencies; its component definition names the image, command, typed inputs, and outputs. section 5.1.1
- Alternatively, decorate a Python function with @dsl.component. Keep imports needed by the function inside it, and provide third-party packages through the image or packages_to_install. section 7.2.1
- Connect outputs to downstream inputs in an @dsl.pipeline function, then compile and execute the pipeline. Do not assume two components can share a pod’s local files. section 7.4 section 7.2
Listing 5.4 Read component definition file
name: Read Data From MinIO
description: Fetches data from MinIO and outputs as a Dataset.
inputs: #1
- name: minio_host
type: String
description: MinIO host URL.
- name: access_key
type: String
description: MinIO access key.
- name: secret_key
type: String
description: MinIO secret key.
- name: bucket_name
type: String
description: Name of the MinIO bucket.
- name: file_name
type: String
description: Name of the file to fetch from MinIO.
outputs: #2
- name: data_output
type: Dataset
description: Output dataset artifact.
implementation:
container:
image: 'varunmallya/read-minio-data:latest' #3
command:
- python3
- /app/src/read_data/read_data.py #4
- --minio_host
- {inputValue: minio_host}
- --access_key
- {inputValue: access_key}
- --secret_key
- {inputValue: secret_key}
- --bucket_name
- {inputValue: bucket_name}
- --file_name
- {inputValue: file_name}
- --data_output_path
- {outputPath: data_output}
The MinIO read-component definition shows the contract between a container command, string inputs, and a Dataset output. section 5.1.1
Batch inference from four components
Read user IDs from MinIO, retrieve their features from Feast, load a model for inference, and write predictions back to MinIO. Intermediate datasets let separate steps pass results forward; the model and features come from their respective services. section 5.1.1
Keep components independent and generic enough to reuse, test, and maintain separately. The same pattern extends beyond inference: dataset download and splitting feed training, and training outputs feed validation. Replacing notebook-only training with components also makes execution more consistent across environments and easier to version and monitor. section 5.2 section 8.1.5 section 8.1
