The gblinear booster configures XGBoost with regularized linear models rather than decision trees. It sums chained linear models, producing a coefficient for each feature and an interpretable linear-form prediction. However, its coefficients and intercept do not have the same interpretation as those from ordinary linear regression or generalized linear models because the booster’s optimization, learning rate, and initial estimate affect them. section 5.4.1
Choose gblinear when a linear-form model suits the problem, while accounting for these practical considerations:
- Use regularization to control the linear model; the end-to-end tuning example evaluates gblinear with both L1 and L2 regularization. section 7.2.5
- Treat missing values carefully: gblinear interprets them as zero, which becomes the feature average after standardization. section 6.1.2
- Avoid reading its coefficients as direct linear-regression effects; the compact linear form improves interpretability without making the coefficients classically interpretable. section 5.4.1
Tune gblinear for tabular regression
An Optuna objective can compare gblinear with gbtree while varying the objective, estimator count, learning rate, sampling rates, and regularization settings. Each trial creates an XGBoost Regressor and uses cross-validation to minimize mean absolute error; objective-specific parameters are adjusted when gblinear is selected. section 7.2.5
Listing 7.17 Optimizing the XGBoost regressor
import optuna
def objective(trial): #1
params = { #2
'booster': trial.suggest_categorical(
'booster',
['gbtree', 'gblinear']
),
'objective': trial.suggest_categorical(
'objective',
['reg:squarederror', 'reg:gamma', 'reg:tweedie']
),
'n_estimators': trial.suggest_int(
'n_estimators', 100, 1000
),
'learning_rate': trial.suggest_float(
'learning_rate', 0.01, 1.0, log=True
),
'subsample': trial.suggest_float(
'subsample', 0.3, 1.0
),
'colsample_bytree': trial.suggest_float(
'colsample_bytree', 0.3, 1.0
),
'max_depth': trial.suggest_int('max_depth', 1, 7),
'min_child_weight': trial.suggest_int('min_child_weight', 1, 7),
'reg_lambda': trial.suggest_float(
'reg_lambda', 1e-9, 100.0, log=True
),
'reg_alpha': trial.suggest_float(
'reg_alpha', 1e-9, 100.0, log=True
),
}
if params['booster'] == 'gblinear': #3
keys_to_remove = [
"colsample_bytree", "max_depth",
"min_child_weight", "subsample"
]
params = {
key:value for key, value in params.items()
if key not in keys_to_remove
}
if params['objective'] == 'reg:tweedie': #4
# Must be between in range [1, 2) : 1=poisson 2=gamma
params['tweedie_variance_power'] = trial.suggest_float(
'tweedie_variance_power', 1.01, 1.99
)
xgb = XGBRegressor(**params) #5
model_pipeline = Pipeline(
[('processing', column_transform),
('xgb', xgb)]
)
cv_splits = cv.split(X, y=X["neighbourhood_more_than_30"])
cv_scores = cross_validate(
estimator=model_pipeline,
X=X,
y=y,
scoring='neg_mean_absolute_error',
cv=cv_splits
) #6
cv_evaluation = np.mean(
np.abs(cv_scores['test_score'])
) #7
return cv_evaluation #8
sqlite_db = "sqlite:///sqlite.db"
study_name = "optimize_XGBoost_tokyo_airbnb"
study = optuna.create_study(
storage=sqlite_db,
study_name=study_name,
direction="minimize",
load_if_exists=True
) #9
study.optimize(objective, n_trials=100) #10
print(study.best_value) #11
print(study.best_params) #12
Inspect the runnable Optuna objective that searches across gblinear and gbtree configurations for an XGBoost regressor. section 7.2.5
