Bayesian optimization tunes expensive model configurations by learning from earlier evaluations. Instead of testing every hyperparameter combination, it builds a surrogate function—a model of the model—and uses that proxy to choose the next trial. section 6.3.4
Each decision balances exploration and exploitation. Exploration tests uncertain regions that might improve the model; exploitation tests configurations already estimated to perform well. An acquisition function scores this promise and uncertainty, while new objective-function results update the surrogate through a prior-to-posterior learning cycle. section 6.3.4
A practical Bayesian-optimization workflow is:
- Define an objective function that proposes hyperparameters, trains the model, and returns a cross-validated metric. section 7.2.5
- Specify search spaces for continuous, integer, categorical, and logarithmic parameters using trial suggestions. section 6.3.4 section 7.2.5
- Run a bounded number of adaptive trials, then inspect the best value, selected parameters, optimization history, and estimated parameter importance. section 6.3.4 section 7.2.5
Tuning an XGBoost regressor
For Tokyo Airbnb price prediction, Optuna evaluated XGBoost boosters, objectives, tree depth, estimator count, learning rate, sampling rates, and regularization. Across 60 trials, it reduced MAE from the initial model's 7,227.435 to 6,616.859 and selected a gbtree model with a Tweedie objective, 901 estimators, depth seven, and learning rate about 0.069. section 7.2.4 section 7.2.5
Listing 7.17 Optimizing the XGBoost regressor
import optuna
def objective(trial): #1
params = { #2
'booster': trial.suggest_categorical(
'booster',
['gbtree', 'gblinear']
),
'objective': trial.suggest_categorical(
'objective',
['reg:squarederror', 'reg:gamma', 'reg:tweedie']
),
'n_estimators': trial.suggest_int(
'n_estimators', 100, 1000
),
'learning_rate': trial.suggest_float(
'learning_rate', 0.01, 1.0, log=True
),
'subsample': trial.suggest_float(
'subsample', 0.3, 1.0
),
'colsample_bytree': trial.suggest_float(
'colsample_bytree', 0.3, 1.0
),
'max_depth': trial.suggest_int('max_depth', 1, 7),
'min_child_weight': trial.suggest_int('min_child_weight', 1, 7),
'reg_lambda': trial.suggest_float(
'reg_lambda', 1e-9, 100.0, log=True
),
'reg_alpha': trial.suggest_float(
'reg_alpha', 1e-9, 100.0, log=True
),
}
if params['booster'] == 'gblinear': #3
keys_to_remove = [
"colsample_bytree", "max_depth",
"min_child_weight", "subsample"
]
params = {
key:value for key, value in params.items()
if key not in keys_to_remove
}
if params['objective'] == 'reg:tweedie': #4
# Must be between in range [1, 2) : 1=poisson 2=gamma
params['tweedie_variance_power'] = trial.suggest_float(
'tweedie_variance_power', 1.01, 1.99
)
xgb = XGBRegressor(**params) #5
model_pipeline = Pipeline(
[('processing', column_transform),
('xgb', xgb)]
)
cv_splits = cv.split(X, y=X["neighbourhood_more_than_30"])
cv_scores = cross_validate(
estimator=model_pipeline,
X=X,
y=y,
scoring='neg_mean_absolute_error',
cv=cv_splits
) #6
cv_evaluation = np.mean(
np.abs(cv_scores['test_score'])
) #7
return cv_evaluation #8
sqlite_db = "sqlite:///sqlite.db"
study_name = "optimize_XGBoost_tokyo_airbnb"
study = optuna.create_study(
storage=sqlite_db,
study_name=study_name,
direction="minimize",
load_if_exists=True
) #9
study.optimize(objective, n_trials=100) #10
print(study.best_value) #11
print(study.best_params) #12
Use the Optuna listing as a runnable pattern for defining a trial objective, suggesting XGBoost parameters, evaluating cross-validated MAE, and printing the best configuration. section 7.2.5
