liveBook · Manning

Data aggregation

As explained in Outlier Detection in Python

Aggregation is the process of combining multiple observations into summary values according to specified grouping dimensions and time intervals.

Data aggregation combines many records into summary values for chosen grouping dimensions and time intervals. Grouping purchases by date, for example, produces one row per day with means, sums, and counts for measures such as quantity, unit cost, total cost, and purchase count. section 12.6.3

Aggregation makes irregular event records comparable across entities or periods. Purchases can be grouped by purchaser and hour or day, while sales and purchases can be resampled into daily time series; missing purchaser-period combinations may receive zero counts. section 17.1

Choose the grouping and summary carefully:

  • Use enough observations within each group and enough groups for comparison; sparse summaries make findings less reliable. section 12.6.3
  • Compare alternative dimensions such as date, day of month, day of week, or hour rather than assuming one grouping reveals every pattern. section 12.6.3
  • Treat unusual aggregate values as investigation leads, not automatic anomalies; calendar composition can explain effects such as an unusually low 31st day. section 12.6.3

Daily purchase summaries

Grouping the purchases table by Date and aggregating Count, Unit Cost, and Total Cost with mean and sum, alongside Purchase ID with count, creates a daily comparison table. Plotting total purchases by day of month can reveal a notably high 28th day, which warrants follow-up rather than immediate classification as an outlier.

How Outlier Detection in Python covers it

How deep it goes
Provides implementation patterns and interpretation cautions, but stops short of a general theory of aggregation.
Examples
Uses daily purchase summaries, day-of-month comparisons, and daily sales-versus-purchases time series.
What it assumes
Assumes tabular transaction records with grouping fields, measures, and usable date or datetime values.
What you can do after
Apply it via pandas groupby and resample to build grouped tables and regular daily time series.
How it connects
Connects aggregation to collective outliers, point anomalies, panel data, and time-series data.

Also explained in

Compare how 19 books explain data aggregation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Outlier Detection in Python. The section links point to the parts of the book it draws on.