HistGradientBoostingClassifier

As explained in Ensemble Methods for Machine Learning

HistogramGradientBoostingClassifier is a scikit-learn classifier that accelerates gradient-boosted decision-tree training by using histogram-based feature processing.

HistGradientBoostingClassifier trains gradient-boosted decision-tree ensembles faster by replacing exact split searches with histogram-based feature binning. It is especially useful when data sets are large enough for this speed–exactness tradeoff to matter. section 5.2 section 5.2.4

Its training process follows this practical sequence:

• Build weak decision-tree learners sequentially, using gradient boosting to approximate the training objective’s gradient at each iteration. section 5.2
• Compress continuous feature values into bins, then evaluate splits between neighboring bins rather than between every pair of successive data points. section 5.2.4
• Control model complexity and training length with parameters such as max_depth, max_iter, and learning_rate; deeper trees can overfit. section 5.2.3

Breast-cancer classification

Using scikit-learn, fit HistGradientBoostingClassifier(max_depth=2, max_iter=20, learning_rate=0.75) on training data, predict the test set, and calculate the error. A reported test error on the breast-cancer data set is 4.2%. section 5.2.4

Figure 5.17 Left: A simple 1D regression problem with 50 data points. Center: Standard tree learning evaluates every possible split, which is illustrated by a line between each pair of data points. The best split is the one with the lowest split criterion (here, squared loss). Right: Histogram-based binning first puts the data into five buckets, and then evaluates the splits between each pair of data buckets. Again, the best split is the one with the lowest split criterion (also squared loss).

Compare exact tree splitting with five-bin histogram splitting to see why fewer candidate thresholds improve scalability. section 5.2.4

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains binning and split reduction procedurally, but stops short of implementation internals and formal derivations.

Examples

Uses breast-cancer classification, five-bin split diagrams, and a scikit-learn training-and-error calculation.

What it assumes

Assumes familiarity with classifiers, decision trees, gradient boosting, train-test evaluation, and learning-rate or complexity parameters.

What you can do after

Apply it via scikit-learn to train, predict, and evaluate a histogram-based gradient-boosting classifier.

How it connects

Connects gradient boosting with decision trees, AdaBoost, histogram-based binning, and LightGBM.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• histogram-based gradient boostingimplements · Histogram-based gradient boosting is gradient boosting that bins feature values to accelerate split finding and reduce memory use.
• gradient boostingimplements · Gradient boosting is an ensemble-learning technique that sequentially adds predictive models, typically weak learners, to reduce a loss by fitting its gradient.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.