Optuna is an open-source hyperparameter optimization framework for adaptively exploring model configurations through repeated trials. It uses tree-structured Parzen estimators and integrates with TensorFlow, PyTorch, and Scikit-learn. section 6.3.4 section 7.2.5
A typical Optuna workflow has four practical parts:
- Define an objective function that accepts a trial and returns an evaluation metric. section 6.3.4
- Use trial.suggest_... methods to select integer, floating-point, or categorical hyperparameters. section 7.2.5
- Evaluate each configuration with cross-validation, a sample test, or another procedure chosen inside the objective. section 6.3.4
- Create a study, choose whether to maximize or minimize the objective, and run a specified number of trials. section 6.3.4
Tuning an XGBoost regressor
For Tokyo Airbnb price prediction, Optuna tests booster types, objectives, tree parameters, learning rate, sampling ratios, and regularization. Each trial trains an XGBoost regressor and uses cross-validation to minimize MAE. After 60 trials, the reported configuration uses gbtree, 901 estimators, depth seven, and a Tweedie objective, with a reduced MAE compared with the initial attempt. section 7.2.5
Listing 7.17 Optimizing the XGBoost regressor
import optuna
def objective(trial): #1
params = { #2
'booster': trial.suggest_categorical(
'booster',
['gbtree', 'gblinear']
),
'objective': trial.suggest_categorical(
'objective',
['reg:squarederror', 'reg:gamma', 'reg:tweedie']
),
'n_estimators': trial.suggest_int(
'n_estimators', 100, 1000
),
'learning_rate': trial.suggest_float(
'learning_rate', 0.01, 1.0, log=True
),
'subsample': trial.suggest_float(
'subsample', 0.3, 1.0
),
'colsample_bytree': trial.suggest_float(
'colsample_bytree', 0.3, 1.0
),
'max_depth': trial.suggest_int('max_depth', 1, 7),
'min_child_weight': trial.suggest_int('min_child_weight', 1, 7),
'reg_lambda': trial.suggest_float(
'reg_lambda', 1e-9, 100.0, log=True
),
'reg_alpha': trial.suggest_float(
'reg_alpha', 1e-9, 100.0, log=True
),
}
if params['booster'] == 'gblinear': #3
keys_to_remove = [
"colsample_bytree", "max_depth",
"min_child_weight", "subsample"
]
params = {
key:value for key, value in params.items()
if key not in keys_to_remove
}
if params['objective'] == 'reg:tweedie': #4
# Must be between in range [1, 2) : 1=poisson 2=gamma
params['tweedie_variance_power'] = trial.suggest_float(
'tweedie_variance_power', 1.01, 1.99
)
xgb = XGBRegressor(**params) #5
model_pipeline = Pipeline(
[('processing', column_transform),
('xgb', xgb)]
)
cv_splits = cv.split(X, y=X["neighbourhood_more_than_30"])
cv_scores = cross_validate(
estimator=model_pipeline,
X=X,
y=y,
scoring='neg_mean_absolute_error',
cv=cv_splits
) #6
cv_evaluation = np.mean(
np.abs(cv_scores['test_score'])
) #7
return cv_evaluation #8
sqlite_db = "sqlite:///sqlite.db"
study_name = "optimize_XGBoost_tokyo_airbnb"
study = optuna.create_study(
storage=sqlite_db,
study_name=study_name,
direction="minimize",
load_if_exists=True
) #9
study.optimize(objective, n_trials=100) #10
print(study.best_value) #11
print(study.best_params) #12
Inspect a runnable objective function that searches XGBoost boosters, objectives, estimator counts, learning rates, sampling ratios, depth, and regularization. section 7.2.5
