A Hessian matrix contains second-order partial derivatives of a scalar function. In optimization, it describes local curvature and complements the gradient by showing how quickly the function changes. section 5.4.2
Newton-style boosting uses Hessian values to adjust gradient updates. Larger curvature reduces the gradient's effect, acting as a brake; smaller curvature accelerates learning. This creates an adaptive learning rate for each training example, although calculating second derivatives can require substantial computation. section 5.4.2
Practical considerations when using Hessians in boosting include:
- For squared-error objectives, the Hessian may be identical for every example and provide little additional benefit. section 5.4.2
- For other objectives, curvature information can improve the direction and rate of optimization updates. section 5.4.2
- Regularization can be added to the Hessian term to balance convergence acceleration against overfitting. section 5.4.2
Newtonian gradient boosting
A custom NewtonianGradientBoosting implementation calculates gradient and Hessian values, then fits a decision tree to -gradient / (Hessian + reg_lambda). The example sets reg_lambda to 0.25 and reports an accuracy of 0.82514. section 5.4.2
Listing 5.13 How XGBoost works
class NewtonianGradientBoosting(GradientBoosting): #1
"""the Newton-Raphson method is used to update the predictions"""
reg_lambda = 0.25 #2
def hessian(self, y_true, y_pred):
hessian = np.ones_like(y_true) #3
return hessian
def fit(self, X, y):
self.init = self.logit(np.mean(y))
y_pred = self.init * np.ones((X.shape[0],))
for k in range(self.n_estimators):
gradient = self.gradient(self.logit(y), y_pred)
hessian = self.hessian(self.logit(y), y_pred)
tree = DecisionTreeRegressor(**self.params)
tree.fit(
X,
-gradient / (
hessian + self.reg_lambda
)
) #4
self.trees.append(tree)
y_pred += self.learning_rate * tree.predict(X)
cls = NewtonianGradientBoosting(n_estimators=300,
learning_rate=0.1,
max_depth=4,
min_samples_leaf=3,
random_state=0) #5
cls.fit(X, y) #6
preds = cls.predict(Xt)
score = accuracy_score(y_true=yt, y_pred=preds)
print(f"Accuracy: {score:0.5f}") #7
Inspect the implementation to see where Hessian calculation enters the fitting loop and how it scales the gradient update. section 5.4.2
