liveBook · Manning

Colsample_bytree

As explained in Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

colsample_bytree is a tree-ensemble hyperparameter specifying the fraction of features considered when splitting each tree.

colsample_bytree controls the fraction of available features considered when splitting each tree in a tree-ensemble model. Using a value below 1 means that individual trees can receive different feature subsets instead of using every predictor. section 5.4.1

The parameter offers a practical trade-off. Considering fewer features can reduce training time and limit overfitting or dependence on individual predictors, but discarding features can also prevent the model from achieving its best results. section 6.4.3 section 5.4.1

Reducing features in a customer model

Suppose a gradient-boosted model uses many customer predictors and takes too long to train. Setting colsample_bytree below 1 allows each tree to consider only part of the feature set, potentially shortening training and encouraging trees to rely on different predictors. Compare the resulting validation performance with the full-feature setting before keeping the change. section 6.4.3 section 5.4.1

Use colsample_bytree when you need to balance model generalization, feature influence, and training cost:

  • Keep the value at 1 when every tree should consider the complete predictor set. section 5.4.1
  • Test values below 1 when overfitting, excessive influence from individual features, or long training times are concerns. section 6.4.3 section 5.4.1
  • Compare feature sampling with GPU acceleration when training time is the main constraint, since reducing features may not maximize model quality. section 6.4.3

How Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI covers it

How deep it goes
Defines the feature fraction and its trade-offs, but stops short of prescribing an optimal value for a particular dataset.
Examples
Uses XGBoost and LightGBM training-time scenarios involving many features and different feature subsets across trees.
What you can do after
Apply it via XGBoost or LightGBM and compare training speed and generalization after testing feature fractions below 1.
How it connects
Connects feature sampling with decision trees, gradient boosting, subsampling, overfitting, underfitting, and GPU-based training.
Cover of Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
liveBookmanning.com
Read Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI. The section links point to the parts of the book it draws on.