XGBoost, short for eXtreme Gradient Boosting, is an open-source framework for training sequential ensembles of regression trees. Its Newton-boosting approach uses weighted residuals to focus successive trees on poorly handled examples, while parallel and distributed processing make it suitable for very large data sets. It supports classification, regression, ranking, and custom loss functions. section 6.3 section 6.6
The “extreme” part combines three concerns: Newton boosting, regularized objectives, and efficient tree learning. The objective fits the data while regularization constrains leaf values and the number of leaves. Limiting leaves encourages shallower, weaker trees and can improve generalization, but introduces hyperparameters that require selection. section 6.3.1 section 6.6
Train an XGBoost model as a controlled iterative process:
- Choose an objective and evaluation metric appropriate to the task; for multiclass classification, softmax can train the model while multiclass error evaluates it. section 6.5.2
- Control each tree’s contribution with learning_rate and tree complexity with max_depth, min_child_weight, feature sampling, and regularization parameters. section 6.5.2
- Use randomized cross-validation rather than exhaustive grid search when the parameter space is expensive to explore. section 6.5 section 7.4.4
- Use early stopping when evaluation performance stops improving, or choose a conservative learning rate proactively to reduce overfitting. section 6.4 section 7.4.4
Document retrieval classifier
For a three-class document-retrieval task, an XGBClassifier uses objective='multi:softmax' and eval_metric='merror'. Setting n_jobs=-1 enables all available CPU cores. A five-fold RandomizedSearchCV can tune learning rate, depth, leaf constraints, feature sampling, and L1/L2 regularization without evaluating every parameter combination. section 6.5.2
xgb = XGBClassifier(booster='gbtree', objective='multi:softmax',
eval_metric='merror', use_label_encoder=False,
n_jobs=-1)
This classifier setup shows the essential task choices: tree boosting, a multiclass softmax objective, multiclass-error evaluation, and parallel CPU training. section 6.5.2
Loss-function choice matters because Newton boosting requires second derivatives. XGBoost therefore does not directly provide MAE, Huber, or standard Huber loss objectives; pseudo-Huber provides a differentiable approximation. For continuous or count targets, available examples include squared-error, pseudo-Huber, Poisson, and Tweedie objectives. section 7.3.2 section 7.4.4
In document retrieval, XGBoost can be compared with LightGBM while tuning on large data through randomized cross-validation. More broadly, performance depends on matching the loss function to the prediction problem and systematically selecting parameters, rather than treating boosting as a one-setting solution. section 6.5 section 7.4.4
