The Newton-Raphson method refines model predictions by combining the first derivative, or gradient, with the second derivative, or Hessian. In gradient boosting, this second-order information adjusts each update to the objective function’s curvature rather than relying on gradient size alone. section 5.4.2
For practical machine-learning work, the method provides several useful behaviors:
- Larger curvature reduces the gradient’s effect, acting as a brake on the learning rate; smaller curvature accelerates it. section 5.4.2
- A regularization term can stabilize the update and help prevent overfitting. section 5.4.2
- XGBoost uses Newton Descent as one of its optimization techniques, alongside specialized feature-splitting and sparse-data handling. section 5.4
A Newton-style boosting update
A custom NewtonianGradientBoosting implementation calculates gradient and Hessian values, then fits a decision tree to -gradient / (hessian + reg_lambda). With reg_lambda set to 0.25, the denominator regularizes the second-order update. section 5.4.2
Listing 5.13 How XGBoost works
class NewtonianGradientBoosting(GradientBoosting): #1
"""the Newton-Raphson method is used to update the predictions"""
reg_lambda = 0.25 #2
def hessian(self, y_true, y_pred):
hessian = np.ones_like(y_true) #3
return hessian
def fit(self, X, y):
self.init = self.logit(np.mean(y))
y_pred = self.init * np.ones((X.shape[0],))
for k in range(self.n_estimators):
gradient = self.gradient(self.logit(y), y_pred)
hessian = self.hessian(self.logit(y), y_pred)
tree = DecisionTreeRegressor(**self.params)
tree.fit(
X,
-gradient / (
hessian + self.reg_lambda
)
) #4
self.trees.append(tree)
y_pred += self.learning_rate * tree.predict(X)
cls = NewtonianGradientBoosting(n_estimators=300,
learning_rate=0.1,
max_depth=4,
min_samples_leaf=3,
random_state=0) #5
cls.fit(X, y) #6
preds = cls.predict(Xt)
score = accuracy_score(y_true=yt, y_pred=preds)
print(f"Accuracy: {score:0.5f}") #7
Inspect the implementation showing initialization, gradient and Hessian calculation, regularization, and tree fitting. section 5.4.2
