Data aggregation combines individual values into summarized groups or totals. In machine-learning delivery, it turns distributed data assets into simpler, consumable records for exploration, training, testing, and operational reuse. section 5.4
Use aggregation to make data and evaluation more manageable:
- Combine data from containers, databases, data marts, and warehouses into a maintainable table or stream. section 5.4
- Aggregate comparable model-test results with means and standard deviations before selecting a model. section 8.3.2
- Preserve checkpoints so current data can support active work while historical results remain reproducible. section 5.4
- Pair aggregation with transformations and rebalancing when source data must become suitable for algorithms. section 5.4
Comparing models across countries
Evaluate two models across 40 equally important countries for young adults. Aggregate each model’s results into a mean and standard deviation, then prefer the model whose overall performance is clearly differentiated rather than merely higher by mean score. section 8.3.2
Figure 5.1 The processing requirements delivered using the dev environment data pipeline infrastructure. This provides the data resources for the ML modeling team in sprint 2 and beyond.
Use the pipeline infrastructure view to connect aggregation with the data resources delivered to modeling work. section 5.4
