Migration uses extract-transform-load (ETL) to move data from source systems through structured processing into a target workflow. In ML operations, reliable data pipelines provide the foundation for repeatable, reproducible, and observable workflows. section 7.1.5 section 2.2
A production-oriented migration connects several operational concerns:
- Extract and curate data while preserving relevance, quality, diversity, representative distributions, and lineage. section 2.1.1
- Transform data through a structured ETL process that can participate in repeatable and reproducible workflows. section 7.1.5
- Load the resulting data into an ML workflow whose components can be observed and maintained alongside other platform services. section 2.2 section 7.1.5
Versioned ML data pipeline
A team collects enterprise data, tracks raw and intermediate dataset versions, applies structured ETL, and connects the result to Kubeflow Pipelines for repeatable experiments. section 2.1.1 section 7.1.5
