ETL is a data-integration process that extracts data from sources, transforms it into a usable structure, and loads it into a target system or data store. In machine-learning projects, it supplies reproducible, current data resources for modeling while exposing source, quality, ownership, security, and privacy constraints. section 2.4 section 5.4
Treat ETL as a delivery workstream with explicit investigation and control points:
- Inventory sources, schemas, update rates, volumes, platforms, owners, and access skills before estimating delivery effort. section 2.4
- Ingest files, streams, or query results from one or more databases and infrastructures into a processing environment. section 5.4
- Transform and integrate the data into intermediate or final tables suitable for creating models. section 5.4
- Document manual preparation steps, identify source-data problems, and test whether supplied samples can be reproduced. section 2.4 section 5.4
- Record security and privacy constraints before choosing where sensitive data should be processed. section 2.4 section 5.4
Cross-system training data
A pipeline can ingest CSV, XLSX, or Parquet files, consume streams such as RabbitMQ or Kafka, and use SQL queries to combine tables from multiple databases. It may move data from cloud, SaaS, or on-premises systems into a target infrastructure, unless security requirements require sensitive data to remain on premises. section 5.4
Figure 5.1 The processing requirements delivered using the dev environment data pipeline infrastructure. This provides the data resources for the ML modeling team in sprint 2 and beyond.
Use the pipeline overview to trace how ingestion and processing deliver data resources to the modeling team. section 5.4
