Data aggregation combines observations into summary values for groups or time periods, making patterns easier to interpret and communicate. section 3.2.1
Aggregation connects technical analysis to stakeholder questions by providing views at useful granularities, such as weeks or months, rather than exposing every individual observation. section 3.2.1
Common aggregation choices include:
- Sum values to show total volume, such as total monthly rat sightings. section 3.2.1
- Calculate the mean or median to summarize typical values and compare the resulting patterns. section 3.2.1
- Choose daily, weekly, monthly, or quarterly periods according to response needs, noise, seasonality, and forecasting goals. section 7.2.1
Reliable aggregation requires consistent measurement units, regular collection intervals, sufficiently long observation periods, and accurate data. Duplicate records, invalid values, or changing schemas can distort metric values. section 7.2.1
Monthly rat sightings
Daily New York City rat-sighting reports contain dense weekly spikes. Grouping them by month and calculating sums, means, and medians reduces that noise and reveals seasonal patterns associated with colder and warmer months. section 3.2.1
rats = rats.reset_index()
rats["day"] = pd.to_datetime(rats["day"], format="%m/%d/%y") #1
rats_group = rats.groupby(pd.Grouper(key="day", axis=0, freq="ME")).agg(
["sum", "mean", "median"]
) #2
rats_group.columns = rats_group.columns.get_level_values(1) #3
plt.plot(rats_group["sum"]) #4
plt.xlabel("Month (Starting 1/1/2018)")
plt.ylabel("# of Rat Sightings")
A pandas workflow converts dates, groups observations by month, and calculates sum, mean, and median values for visualization. section 3.2.1
