Tree pruning

As explained in Grokking Machine Learning

Tree pruning is the removal of decision-tree branches that fail a specified split-quality condition.

Tree pruning removes branches that fail a specified split-quality condition. In XGBoost, it helps prevent overfitting: a split is kept only when its similarity gain exceeds the minimum split-loss threshold. section 12.5

Pruning by similarity gain

For an example tree, only splits with similarity gain higher than 1 are allowed. One split fails that condition, so removing it produces the pruned tree used as a weak learner. section 12.5

Figure 12.24 On the left, we have the same tree from figure 12.23, with an extra piece of information: the similarity gain. We obtain this by subtracting the similarity score for each node from the combined similarity score of the leaves. We only allow splits with a similarity gain higher than 1 (our minimum split loss hyperparameter), so one of the splits is no longer permitted. This results in the pruned tree on the right, which now becomes our weak learner.

Compare the candidate tree and its similarity gains with the resulting pruned tree. section 12.5

How Grokking Machine Learning covers it

How deep it goes

Explains pruning through similarity gain and a minimum split-loss threshold, with a worked example using a threshold of 1.

Examples

Uses an engagement-prediction tree to show a split rejected when its similarity gain does not exceed 1.

What it assumes

Relates pruning to trees fitted to residuals and introduces similarity scores before applying the pruning condition.

What you can do after

Evaluate whether a candidate tree split survives by comparing its similarity gain with the minimum split-loss threshold.

How it connects

Connects decision trees and gradient boosting with XGBoost, residuals, and similarity scores.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Related concepts

• overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.