liveBook · Manning

Data aggregation

As explained in Digital Twins in Action

An aggregate is a summary value computed from multiple data records, such as an average, minimum, or maximum.

Data aggregation computes summary values from multiple records. For time-series data, readings are grouped into time windows, then reduced to statistics such as a mean, minimum, maximum, or sum. A digital twin can use those summaries to show trends without repeatedly processing every sensor reading. section 4.3.3 section 6.3.2

From readings to a useful chart

A temperature sensor produced 6,375 readings over 45 days. Calculating each day’s low, high, and mean turns them into 45 daily records. Plotting those records makes the spring warming trend and unusual patterns easier to see than either the raw readings or the summary table alone. section 6.2.1

timestamp,min,max,mean
1754784000,13.8,18.4,16.31
1754870400,12.8,17.6,15.57
1754956800,13.3,17.4,15.99
1755043200,15.8,19.3,18.14
.
. 40 readings
.
1758585600,19.9,22.7,21.64

The daily records show the time bucket alongside its minimum, maximum, and mean. Each row summarizes many readings rather than representing one sensor measurement. section 6.2.1

Choose windows to match the question. Minute-level summaries can support close inspection; hourly, daily, or monthly summaries can support longer views. Precomputing these resolutions avoids recalculating statistics for every dashboard request and lets a chart switch resolution as its zoom level or viewport changes. In a data warehouse, these resolutions are called grains. section 4.3.3 section 6.3.2

Aggregation is not the same as decimation. Aggregation calculates new values for time buckets and suits long-term trend dashboards. Decimation removes selected points while retaining the curve’s shape and suits zoomed-in, high-frequency views. A mean alone can obscure a brief extreme; storing the minimum and maximum alongside it retains evidence of peaks within each window. section 6.3.2

Aggregation can also happen as data arrives. A 30-second sliding average of one-second temperature readings drops the oldest reading as each new one enters, smoothing noise without waiting for a batch. The same windowing approach can track the share of late train movements over five minutes for a timely operational measure. section 4.4.2

For a stored dashboard dataset, aggregate during transformation instead. One Apache Spark example reads minute-by-minute power CSV data, groups records by the week of their local timestamp, sums watt-hours into kilowatt-hours, and sums cost. Validate readings and standardize units before relying on derived figures; retain transformation lineage so results can be traced back to their inputs. section 4.4.4

How Digital Twins in Action covers it

How deep it goes
Explains time-window summaries, stored resolutions, and sliding windows, with a Spark grouping example rather than aggregation internals.
Examples
Uses daily temperature summaries, a sliding temperature average, train-movement rates, and weekly power-consumption totals.
What it assumes
Assumes familiarity with timestamped sensor readings; defines aggregation through concrete summaries rather than requiring statistical theory.
What you can do after
Apply it via Apache Spark to produce weekly consumption and cost totals, or choose time-window summaries for responsive charts.
How it connects
Links aggregation to time-series databases, data visualization, decimation, streaming ingestion, and ETL.

Also explained in

Compare how 19 books explain data aggregation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Digital Twins in Action. The section links point to the parts of the book it draws on.