The learning rate controls how much each new weak learner changes a boosted ensemble. It scales each successive contribution, so it governs how quickly ensemble complexity grows rather than changing the learner itself. section 6.4.1 section 5.2.3
Think of it as the size of each step in sequential learning. A slower rate makes updates more conservative and commonly requires more iterations; in gradient boosting, this can make successive learners more robust to outliers and noise. A larger rate makes fitting more aggressive, which can reduce training error while risking overfitting. section 5.2.3 section 5.4.1 section 6.4.1
Tune the learning rate together with the rest of the boosting configuration:
- Balance it with n_estimators. More base learners add more sequential updates; learning_rate constrains their accumulated influence. section 4.2.3 section 4.3.2
- Use cross-validation to compare candidate rates by validation error, not training error alone. In one LightGBM sweep, validation error fell and then rose, with 0.4 best among the tested values. section 5.4.1
- Watch both extremes: very low rates can be too conservative and underfit, while high rates can become aggressive and overfit. section 6.4.1
- In AdaBoost, smaller rates shrink weak-learner weights αt and reduce variation in training-example weights; larger rates have the opposite effect. section 4.2.3
Cross-validating a rate
For a breast-cancer classifier, split data with StratifiedKFold, train a LightGBM model for each candidate learning_rate, and average training and validation errors across folds. Choose the rate with the best validation result rather than the lowest training error. section 5.4.1
Listing 5.4 Cross validation with LightGBM and scikit-learn
from sklearn.model_selection import StratifiedKFold
import numpy as np
n_learning_rate_steps, n_folds = 10, 10 #1
learning_rates = np.linspace(0.1, 1.0, num=n_learning_rate_steps)
splitter = StratifiedKFold( #2
n_splits=n_folds, shuffle=True, random_state=42)
trn_err = np.zeros((n_learning_rate_steps, n_folds)) #3
val_err = np.zeros((n_learning_rate_steps, n_folds)) #3
for i, rate in enumerate(learning_rates): #4
for j, (trn, val) in enumerate(splitter.split(X, y)):
gbm = LGBMClassifier(boosting_type='gbdt', n_estimators=10,
max_depth=1, learning_rate=rate)
gbm.fit(X[trn, :], y[trn])
trn_err[i, j] = (1 - accuracy_score(y[trn], #5
gbm.predict(X[trn, :]))) * 100 #5
val_err[i, j] = (1 - accuracy_score(y[val], #5
gbm.predict(X[val, :]))) * 100 #5
trn_err = np.mean(trn_err, axis=1) #6
val_err = np.mean(val_err, axis=1) #6
This loop tests learning rates from 0.1 to 1.0 across ten stratified folds while holding tree depth and estimator count fixed, isolating the rate’s effect. section 5.4.1
Library names vary, but the role is consistent: LightGBM also accepts shrinkage_rate and eta aliases, while scikit-learn, LightGBM, and XGBoost expose learning_rate directly. Set only one LightGBM alias. For larger parameter spaces, randomized cross-validation can search learning_rate alongside depth, leaf constraints, feature sampling, and regularization. section 5.4.1 section 6.5.2
