A learning rate controls how much a model changes at each fitting step. It is the practical dial between fast, decisive updates and smaller, cautious ones: a rate that is too high can overshoot the target and worsen error, while a low rate is safer but can require substantially more computation. section 4.3.5
For stochastic gradient descent (SGD), each batch produces an update to a linear model’s coefficients. The learning rate scales that update as the model repeatedly sees batches over iterations, or epochs. Because outliers can cause abrupt coefficient changes, gradual updates and averaging can make fitting more stable. section 4.3.5
Streaming a large tabular dataset
Use batch-by-batch fitting when the dataset is too large to process at once. Initialize an SGD classifier with the target labels, call partial_fit on successive batches, and repeat over the data as needed. The learning rate determines how strongly each batch changes the current model, so it affects both convergence behavior and the cost of training. section 4.3.5 section 4.3.5
In gradient boosting, the learning rate scales each tree’s correction to the current ensemble prediction. The model combines a sequence of regression trees, even for binary classification, and the rate determines how strongly each new tree contributes. It therefore works together with n_estimators rather than acting as an isolated setting. section 5.2.1 section 5.4.1
Listing 5.6 Building a gradient boosting classifier
from sklearn.tree import DecisionTreeRegressor
import numpy as np
class GradientBoosting():
def __init__(self, learning_rate=0.1, n_estimators=10, **params):
self.learning_rate = learning_rate
self.n_estimators = n_estimators
self.params = params
self.trees = list()
def sigmoid(self, x):
x = np.clip(x, -100, 100)
return 1 / (1 + np.exp(-x)) #1
def logit(self, x, eps=1e-6):
xp = np.clip(x, eps, 1-eps)
return np.log(xp / (1 - xp)) #2
def gradient(self, y_true, y_pred):
gradient = y_pred - y_true #3
return gradient
def fit(self, X, y):
self.init = self.logit(np.mean(y)) #4
y_pred = self.init * np.ones((X.shape[0],))
for k in range(self.n_estimators):
gradient = self.gradient(self.logit(y), y_pred)
tree = DecisionTreeRegressor(**self.params)
tree.fit(X, -gradient) #5
self.trees.append(tree)
y_pred += (
self.learning_rate * tree.predict(X)
) #6
def predict_proba(self, X):
y_pred = self.init * np.ones((X.shape[0],))
for tree in self.trees:
y_pred += (
self.learning_rate * tree.predict(X)
) #7
return self.sigmoid(y_pred)
def predict(self, X, threshold=0.5):
proba = self.predict_proba(X)
return np.where(proba >= threshold, 1, 0)
Inspect a compact gradient-boosting implementation where learning_rate and n_estimators are explicit constructor settings, then follow the fitted trees as they build the prediction. section 5.2.1
Tune the learning rate with the rest of the boosting configuration:
- Pair a lower learning rate with more trees. Lower rates generally require a higher optimal n_estimators value, while adding trees increases training time. section 5.4.1 section 6.3.5
- Reduce the learning rate when increasing maximum tree depth. Deeper trees add complexity and may overfit; smaller updates provide a more cautious fitting process. section 6.3.5
- Start manual GBDT trials by fixing a plausible tree count, usually 100 to 1,000, then test rates around 2 to 10 divided by that count. For 1,000 trees, test 0.002 to 0.01. section 6.3.5
Choosing a rate for an XGBoost-style ensemble
Suppose a tabular prediction task uses 1,000 trees. Rather than assuming a default is best, hold that tree count fixed during an initial search and compare learning rates from 0.002 through 0.01. Evaluate prediction performance on unseen data, since the best number of trees and its matching learning rate depend on the problem. section 5.4.1 section 6.3.5
A learning rate cannot remove every optimization difficulty. Linear models have a simple bowl-shaped error surface, whereas more complex models can encounter local minima or plateaus depending on their starting point. Treat the rate as one part of a broader optimization and generalization decision that also includes model complexity, number of trees, and evaluation on held-out data. section 4.3.5 section 5.4.1
