A hyperparameter is a configuration value chosen before or outside model fitting. It shapes how training proceeds or how complex the resulting model can become, rather than being learned directly from the training examples. section 5.5.2 section 8.2.3
Common hyperparameters control several practical aspects of training:
- Learning rate controls how quickly ensemble complexity grows. section 5.5.2 section 8.2.3
- Iterations, estimators, leaf nodes, tree depth, and minimum leaf sizes control training duration or estimator complexity. section 5.5.2 section 8.2.3
- Regularization, subsampling, column sampling, and early stopping provide additional ways to limit overfitting. section 5.5.2 section 8.2.3
Rather than guessing one configuration, compare candidates with cross-validation. GridSearchCV exhaustively evaluates a specified grid; RandomizedSearchCV samples fewer combinations when exhaustive search is costly. With refit enabled, the selected configuration trains a final model on all available training data. section 5.5.2 section 4.4.2
CatBoost configuration search
A CatBoost classifier searches depth values 1 and 3, iteration counts 5, 10, and 15, and learning rates 0.01 and 0.1 using five-fold cross-validation. The reported best combination is depth 3, 15 iterations, and learning rate 0.1. Separately, an evaluation set with early_stopping_rounds=5 can stop training when additional iterations no longer provide significant improvement. section 8.2.3
