Data transformation converts raw data into structures, representations, or formats that downstream analysis and machine-learning algorithms can consume. In a pipeline, aggregations and transformations consolidate many data assets into a simpler table or stream while addressing imbalance, malformed records, and changing modeling needs. section 5.4
A maintainable transformation pipeline should:
- Combine distributed source tables into consumable records. section 5.4
- Rebalance uneven item distributions before model training. section 5.4
- Render unstructured data to standard formats or sizes, filter corrupted items, and use augmentation to create training examples. section 5.4
- Preserve current data and reproducible checkpoints while accommodating later manipulations. section 5.4
Figure 5.1 The processing requirements delivered using the dev environment data pipeline infrastructure. This provides the data resources for the ML modeling team in sprint 2 and beyond.
Pipeline infrastructure supplies processed data resources for the modeling team across subsequent sprints. section 5.4
