An object store gives an ML workflow an external place to keep datasets. Files go into a bucket under keys; the workflow can retain their paths instead of relying on files left on a workstation. In one workflow, MinIO holds prepared data while MLflow records the dataset paths for later loading, reproducibility, or debugging. section 4.1.2 section 7.3.1
A practical setup has three steps:
- Create a MinIO bucket. For a local connection to the Kubeflow instance, forward its service with `kubectl port-forward svc/minio-service -n kubeflow 9000:9000`. section 4.1.2
- Prepare data before uploading it. An object-detection workflow downloads an archive, uncompresses it, then copies the prepared images and labels into MinIO. section 7.3.1
- Record external dataset paths alongside an MLflow run. Training, test, and reference data can each have a path containing the run ID. section 4.1.2
Listing 7.13 Initializing a MinIO client and creating a bucket
Configure a boto3 S3 client with the Kubeflow MinIO endpoint, then create the bucket before uploading files. The shown credentials are for this example instance. section 7.3.1
Listing 7.14 Uploading images and labels to MinIO
import os
for f in ["images", "labels"]:
local_dir_path = os.path.join(output_dir, "DATA", f)
files = os.listdir(local_dir_path)
for file in files:
local_path = os.path.join(local_dir_path, file) #1
s3_path = os.path.join(bucket_name, f, file) #2
minio_client.upload_file(
local_path, bucket_name, s3_path) #3
Upload local image and label files with `upload_file`, supplying a bucket and a destination key for each file. section 7.3.1
Keep a training run tied to its data
Save a training CSV under a key such as `income-classifier-datasets/train-<run-id>.csv` in the `mlflow-datasets` bucket. Convert the pandas data frame with `mlflow.data.from_pandas`, supplying its external path, then log it as an MLflow input. The path points back to the stored dataset; logging the reference is not a substitute for uploading the file. section 4.1.2
MinIO offers an S3-like interface for scripts and automation; GCS or Amazon S3 can serve as alternative object stores for the tracked datasets. MinIO also offers versioning, but it may need enabling and has configuration quirks—check it before relying on it for version control. section 4.1.2 section 7.3.1
