liveBook · Manning

Data aggregation

As explained in Timeless Algorithms, The Foundational Ideas

Aggregation is the combination of multiple observations, models, or outputs into a consolidated result.

Data aggregation combines multiple outputs into one result. In ensemble learning, aggregating predictions from different decision trees reduces dependence on any single tree, whose structure may change substantially with its training sample. The individual trees remain unstable; the collective prediction can be more stable. section 9.0 section 9.2.2

In bagging, each tree is trained on a bootstrap sample drawn with replacement from the original observations. For classification, the ensemble predicts the class with the most votes. For regression, it averages the numerical predictions. With B trees, the regression prediction at x is ŷ(x) = (1/B) ∑ᵦ₌₁ᴮ Tᵦ(x), where Tᵦ(x) is tree b’s prediction. Unusually high and low predictions can balance one another. section 9.2.1 section 9.2.2

A vote across trees

If 62 of 100 trees predict one class and 38 predict another, the ensemble selects the class with 62 votes. A single incorrect vote need not determine the result. section 9.2.2 section 9.2.3

Figure 9.1 Bagging with decision trees. Bootstrap sampling creates multiple training sets by drawing observations with replacement, so each sample contains duplicated and omitted observations. A separate decision tree is trained on each sample using the same procedure. The trees’ predictions are then combined by majority vote for classification or averaging for regression, producing a more stable final prediction.

Follow bootstrap samples through separate trees to a vote or average. section 9.2.2

Aggregation works best when trees do not make the same errors. If a strong predictor produces similar early splits across trees, their errors may be correlated, and voting or averaging cannot cancel a mistake shared by most of them. Random forests add random feature selection at each split to encourage diversity; limiting the candidate features can also weaken individual trees, so generalization depends on their strength as well as their correlation. section 9.2.3 section 9.9

Adding trees reduces variability associated with a finite ensemble, but a stable prediction is not necessarily an accurate one. More trees do not correct data leakage, unrepresentative samples, unsuitable settings, or errors shared by the forest’s trees. section 9.4.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains voting and averaging, variance reduction, correlated errors, and the limits of adding trees.
Examples
Illustrates classification voting with 62 versus 38 tree predictions and regression aggregation with an average.
What you can do after
Evaluate how tree diversity and shared errors affect the stability of ensemble predictions.
How it connects
Connects bootstrap sampling, decision trees, random forests, variance, and correlated errors.

Also explained in

Compare how 20 books explain data aggregation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.