Validation error measures a model’s prediction mistakes on data held aside from fitting. It gives a practical check on whether improving training performance is beginning to overfit the training data. section 5.4.1 section 5.4.2
Use validation error to compare settings and monitor training:
- Split the data into training and validation portions, or use stratified cross-validation to preserve class distributions across folds. section 5.4.1
- Fit each candidate model using only its training portion, then calculate its error on the corresponding validation portion. section 5.4.1 section 4.3.2
- Prefer the setting with the lowest validation error when lower error indicates better performance; stop adding estimators when validation performance no longer improves. section 5.4.1 section 5.4.2
Choosing a LightGBM learning rate
Ten-fold stratified cross-validation compares learning rates from 0.1 through 1.0. Training error keeps decreasing as the rate rises, while validation error first decreases and then increases; 0.4 gives the lowest validation error and is selected. section 5.4.1
Monitoring ensemble growth
During boosting, validation performance is checked after each added estimator. With early stopping set to five rounds, training ends when validation performance fails to improve for five iterations; the best recorded iteration is retained. section 5.4.2
Listing 5.4 Cross validation with LightGBM and scikit-learn
from sklearn.model_selection import StratifiedKFold
import numpy as np
n_learning_rate_steps, n_folds = 10, 10 #1
learning_rates = np.linspace(0.1, 1.0, num=n_learning_rate_steps)
splitter = StratifiedKFold( #2
n_splits=n_folds, shuffle=True, random_state=42)
trn_err = np.zeros((n_learning_rate_steps, n_folds)) #3
val_err = np.zeros((n_learning_rate_steps, n_folds)) #3
for i, rate in enumerate(learning_rates): #4
for j, (trn, val) in enumerate(splitter.split(X, y)):
gbm = LGBMClassifier(boosting_type='gbdt', n_estimators=10,
max_depth=1, learning_rate=rate)
gbm.fit(X[trn, :], y[trn])
trn_err[i, j] = (1 - accuracy_score(y[trn], #5
gbm.predict(X[trn, :]))) * 100 #5
val_err[i, j] = (1 - accuracy_score(y[val], #5
gbm.predict(X[val, :]))) * 100 #5
trn_err = np.mean(trn_err, axis=1) #6
val_err = np.mean(val_err, axis=1) #6
Inspect a complete scikit-learn and LightGBM cross-validation setup that records training and validation errors across learning rates and folds. section 5.4.1
