A model cannot train on data it cannot reliably fetch. Extract, transform, load (ETL) moves information from its sources into a usable dataset: fetch it, tailor its structure, and store the result. For example, an external API’s JSON files might become local CSV or Parquet files. The immediate goal is data availability, not an elaborate storage architecture. section 6.2.1
Design for both training and runtime. Training data should be easy to fetch, ideally through one command or call when it combines several sources. The same sources must also be available during inference, although efficient inference-time fetching is a separate design problem. If a dataset is small and unlikely to grow dramatically, familiar storage may be enough; rapid growth deserves more attention to the data model. section 6.2.1
Before building the pipeline, settle a few practical questions:
- Which sources supply the data, and how will the result be represented and stored? section 6.7
- Which samples need filtering, and which corner cases need manual or automatic handling? section 6.7
- How will features and labels be produced? Treat these as related dataset decisions rather than assuming one fixed order of operations. section 6.7 section 6.8
Daily retail aggregates
A demand-forecasting dataset draws on transactions, product and store information, and external data such as weather. One pipeline aggregates transactions daily and adds each new partition to a table of aggregates. Missing or duplicated transaction, stock, and promotion records need filtering before aggregation; purchased external data still needs checks. Existing history helps the forecast learn sales patterns, but the pipeline must keep delivering usable additions. section 6.7.1
Photo-search datasets
Photo descriptions and clicks arise from normal business activity, so their immediate need is appropriate ETL and storage. Clicks can be noisy because of user behavior and scraping bots. A separate labels dataset requires a labeling budget. Keeping relevant user, seller, and session identifiers supports later filtering and feature engineering without aggregating everything at ingestion. section 6.7.2
A successful job status is not proof of a healthy dataset: a job can process zero rows, report success, and leave an upstream failure unnoticed. Check that expected data arrived and remains usable. Pipeline failures, changed formats, extraction queries, and feature-calculation code can all damage downstream model performance. Aim for reproducibility, consistency, and availability—not merely a green status indicator. section 14.3.1 section 6.8
