A Dataset artifact type represents machine-learning datasets and carries them between KFP pipeline components with metadata describing their characteristics and transformations. Use it for raw inputs, processed data, features, splits, and other structured sample collections. section 7.2.3
Typical pipeline data represented as Dataset artifacts includes:
- Training, validation, and test data. section 7.2.3
- Feature sets and preprocessed data. section 7.2.3
- Data splits and other structured collections of samples. section 7.2.3
@dsl.component
def prepare_dataset(raw_data: Input[Dataset],
processed: Output[Dataset]):
df = pd.read_csv(raw_data.path)
df_processed = process_data(df)
df_processed.to_csv(processed.path, index=False)
processed.metadata.update({
'num_samples': len(df_processed),
'columns': list(df_processed.columns),
'preprocessing_date': datetime.now().isoformat()
})
A component consumes a raw Dataset through Input[Dataset], writes processed data through Output[Dataset], and records sample counts, columns, and preprocessing time in metadata. section 7.2.3
Validation pipeline integration
A YOLO validation component accepts a validation Dataset alongside a trained Model, runs model.val(), and emits evaluation Metrics such as mAP values. The Dataset artifact therefore forms the data dependency between preparation, training, and validation. section 8.1.4
