How deeply does Machine Learning Platform Engineering cover MinIO?

Shows bucket creation, client configuration, uploads, port-forwarding, and S3-style references; leaves MinIO versioning details to documentation.

MinIO in Machine Learning Platform Engineering

MinIO is an open-source object-storage server that provides an Amazon S3-compatible interface for storing unstructured data.

MinIO is an object store with an S3-compatible interface. In an ML workflow, it gives datasets and feature files an external location that training runs, feature definitions, and pipeline components can reference. The Kubeflow setup includes a MinIO service; the examples use it rather than requiring a separate cloud object store. section 4.1.2 section 7.3.1

Connect to the service from the environment where your code runs:

From a local workstation, port-forward the Kubeflow MinIO service to port 9000. Create a bucket before writing datasets; the MLflow example uses `mlflow-datasets`. section 4.1.2
From a Kubeflow component, configure an S3 client with the MinIO service endpoint and credentials, then create a bucket and upload files. The object-detection workflow stores prepared images and labels this way. section 7.3.1
For Feast, specify feature files as `s3://` paths and override the S3 endpoint to reach port-forwarded MinIO. The example registers Parquet feature sources and places the Feast registry at an S3 path. section 4.2.1

Keep a training run tied to its data

Save the feature, training, and test data frames in MinIO under paths containing the MLflow run ID. Record those external paths when converting the pandas data frames to MLflow datasets. The stored data can then be loaded for reproducibility or debugging, while MLflow tracks the run’s datasets, parameters, and metrics. section 4.1.2

Listing 7.13 Initializing a MinIO client and creating a bucket

import boto3
minio_client = boto3.client(                           #1
    's3',                                               #1
    endpoint_url='http://minio-service.kubeflow:9000',  #1
    aws_access_key_id='minio',                          #1
    aws_secret_access_key='minio123')                   #1
try: #1
    minio_client.create_bucket(Bucket=bucket_name) #2
    except Exception as e:    pass

The client configuration uses the in-cluster MinIO service address. Its broad exception handler suppresses bucket-creation failures, so do not treat this example as sufficient operational error handling. section 7.3.1

Listing 4.9 Defining the location of demographic feature file

demo_features_parquet_file_source = FileSource(
    file_format=ParquetFormat(),    #1

    path = 
"s3://feature_data_sets/demographic_features.parquet"    #2
    s3_endpoint_override = "http://localhost:9000"    #3
)

The Feast source combines an S3-style file path with a localhost endpoint override, matching the port-forwarded connection rather than the in-cluster address. section 4.2.1

MinIO is the storage layer, not the experiment tracker or feature store: MLflow records dataset references, and Feast defines how stored feature files are used. MinIO versioning is mentioned as an option, but it may need enabling and has configuration quirks. section 4.1.2 section 4.2.1 section 7.3.1

How Machine Learning Platform Engineering covers it

Also explained in

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Platform Engineering. The section links point to the parts of the book it draws on.