A dataset split divides data into separate training, validation, and test sets. For an object detector, the split must keep image files and their labels organized so each set can be used by the next pipeline stage. section 7.3.1
In a Kubeflow data-preparation pipeline, a download component supplies an Input[Dataset] to a split component. The split component takes a random_state and writes train, validation, and test Output[Dataset] artifacts. Its Python 3.11 component installs scikit-learn to use train_test_split. section 7.3.1
Listing 7.20 Splitting the dataset into 75/15/10% for train/val/test
@dsl.component(
packages_to_install=["scikit-learn"],
base_image="python:3.11"
)
def split_dataset(
random_state: int,
input_dataset: Input[Dataset],
train_dataset: Output[Dataset], #1
validation_dataset: Output[Dataset],
test_dataset: Output[Dataset]
):
import os
import glob
import shutil
from sklearn.model_selection import train_test_split
BASE_PATH = "MINIDATA" #2
images = list(
glob.glob(
os.path.join(
input_dataset.path,
"DATASET",
BASE_PATH,
"images", "**")
)
) #3
labels = list(
glob.glob(
os.path.join(
input_dataset.path,
"DATASET",
BASE_PATH,
"labels", "**")
)
)
train_ratio = 0.75 #4
validation_ratio = 0.15
test_ratio = 0.10
x_train, x_test, y_train, y_test = train_test_split( #5
images,
labels,
test_size=1 - train_ratio,
random_state=random_state
)
x_val, x_test, y_val, y_test = train_test_split( #6
x_test,
y_test,
test_size=test_ratio / (test_ratio + validation_ratio),
random_state=random_state
)
The component listing shows a 75% training, 15% validation, and 10% test split, with image and label paths read from the input dataset. section 7.3.1
After splitting, create images and labels directories under each output artifact and copy the selected files into them. This matches the directory layout expected by the YOLOv8 dataset configuration. A downstream component can list output files as a sanity check before training. section 7.3.1
Listing 7.21 Helper function to organize dataset splits
def split_dataset(...):
for dataset_output, x_files, y_files in [ #1
(train_dataset, x_train, y_train),
(validation_dataset, x_val, y_val),
(test_dataset, x_test, y_test)
]:
os.makedirs(
os.path.join(
dataset_output.path,
"images"
),
exist_ok=True
) #2
os.makedirs(
os.path.join(dataset_output.path, "labels"),
exist_ok=True
)
for src in x_files: #3
dest = os.path.join(
dataset_output.path,
"images",
os.path.basename(src)
)
shutil.copy2(src, dest)
for src in y_files: #4
dest = os.path.join(
dataset_output.path,
"labels",
os.path.basename(src)
)
shutil.copy2(src, dest)
The helper shows how each split's selected images and labels are copied into its output artifact. section 7.3.1
Treat the split as a reusable pipeline output, not just a notebook operation. A later training-and-validation pipeline re-downloaded the full dataset and then re-downloaded splits for training and validation; refactoring its data handling with a persistent volume claim addresses that repeated-transfer pattern. section 9.1.1
