K-fold cross-validation (CV) estimates how a model will behave on unseen data without setting aside just one fixed validation split. It partitions a data set into k nonoverlapping subsets, called folds. Each run trains on k−1 folds and validates on the remaining fold, so every example serves once as validation data. The resulting validation results can be compared or averaged when evaluating models and choosing hyperparameters. section 3.3.2 section 5.4.1
Figure 3.15 k-fold CV (here, k=5) generates k different splits of the data set into a training set and a validation set. This simulates out-of-sample validation during training.
The k=5 layout makes the rotation concrete: each fold uses a different fifth of the data for validation and the other four fifths for training. section 3.3.2
A practical k-fold workflow is:
- Split the training data, often randomly, into k equal-sized disjoint subsets. section 3.3.2
- For each fold, fit a fresh model on all other folds and score it on the held-out fold. section 3.3.2
- Compare validation performance across candidate settings, using averaged fold results to select a promising setting. section 5.4.1 section 6.4.1
- Refit a final model on all available training data after selecting the best setting. section 4.4.2
Stratified k-fold splitting preserves class proportions across folds, which is useful for classification data. In scikit-learn, `StratifiedKFold` supplies training and validation indices for each split. For a candidate learning rate, train and validation errors can be collected over all folds and then averaged; comparing those averages exposes both underfitting and overfitting. A validation-error minimum identifies the best choice among the settings tested. section 5.4.1 section 6.4.1
Tuning a boosting learning rate
For breast-cancer classification, test several learning rates over 10 stratified folds. A low rate can make boosting too conservative and underfit; a high rate can make it aggressive and overfit. Select the rate with the lowest averaged validation error rather than the lowest training error. The same pattern works for other learning parameters, including tree depth and leaf-node count. section 6.4.1 section 5.4.1
CV can also prevent a stacking meta-estimator from learning from base-model predictions made on data those base models already saw. For each fold, train each level-1 estimator on the training portion and generate meta-features only for the held-out portion. Assemble these out-of-fold meta-features to train the level-2 estimator, then retrain the level-1 estimators on the full data set for prediction. This increases ensemble diversity and reduces stacking’s tendency to overfit noisy data. section 3.3.2 section 3.5
The cost is repeated training: k folds require k training-and-validation runs for each candidate configuration, and stacking multiplies that work across base estimators. Exhaustive grid search can therefore be inefficient when many combinations are available; randomized search samples fewer combinations. Library-specific helpers also perform CV directly, including LightGBM, XGBoost, and CatBoost interfaces. section 3.5 section 4.4.2 section 8.2.3
