DecisionTreeRegressor is a scikit-learn estimator for fitting regression trees that predict numeric values. In gradient boosting, each regressor learns corrections to add to the predictions produced by earlier trees—even when the overall task is classification. section 5.2.1
Useful tree-complexity parameters include:
- max_depth, which limits how deep the tree can grow. section 5.2.1
- min_samples_split, which sets the minimum samples required to split an internal node. section 5.2.1
- min_samples_leaf, which sets the minimum samples required in a leaf node. section 5.2.1
Listing 5.6 Building a gradient boosting classifier
from sklearn.tree import DecisionTreeRegressor
import numpy as np
class GradientBoosting():
def __init__(self, learning_rate=0.1, n_estimators=10, **params):
self.learning_rate = learning_rate
self.n_estimators = n_estimators
self.params = params
self.trees = list()
def sigmoid(self, x):
x = np.clip(x, -100, 100)
return 1 / (1 + np.exp(-x)) #1
def logit(self, x, eps=1e-6):
xp = np.clip(x, eps, 1-eps)
return np.log(xp / (1 - xp)) #2
def gradient(self, y_true, y_pred):
gradient = y_pred - y_true #3
return gradient
def fit(self, X, y):
self.init = self.logit(np.mean(y)) #4
y_pred = self.init * np.ones((X.shape[0],))
for k in range(self.n_estimators):
gradient = self.gradient(self.logit(y), y_pred)
tree = DecisionTreeRegressor(**self.params)
tree.fit(X, -gradient) #5
self.trees.append(tree)
y_pred += (
self.learning_rate * tree.predict(X)
) #6
def predict_proba(self, X):
y_pred = self.init * np.ones((X.shape[0],))
for tree in self.trees:
y_pred += (
self.learning_rate * tree.predict(X)
) #7
return self.sigmoid(y_pred)
def predict(self, X, threshold=0.5):
proba = self.predict_proba(X)
return np.where(proba >= threshold, 1, 0)
The listing shows DecisionTreeRegressor configured inside a custom gradient boosting classifier, with tree depth and leaf-size controls passed through. section 5.2.1
