Weight decay, also called L2 regularization, penalizes the sum of squared model coefficients. The penalty gives larger coefficients more weight, so optimization favors generally smaller weights rather than eliminating features outright. It is used to reduce model complexity, limit overfitting, and improve generalization when many predictors are available. section 4.3.2 section 4.4
Unlike L1 regularization, which can drive coefficients to zero and implicitly select features, L2 keeps all features in the calculation. With multicollinear predictors, L2 tends to assign similar coefficient values, whereas L1 may retain only one predictive feature. L2 is also described as relatively fast to compute. section 4.3.2 section 4.4
Ridge regression with engineered tabular features
A Ridge model can follow categorical encoding, numeric passthrough, polynomial expansion, and standardization in a pipeline. Set alpha to control L2 regularization, then use cross-validation and RMSE to evaluate the regularized regression model rather than judging it from one split. section 4.3.2
L2 regularization also applies to logistic regression. Solver choice constrains which penalties are available: lbfgs, newton-cg, newton-cholesky, and sag support L2 or no penalty, while saga supports L1, L2, elastic net, or no penalty. Some large-data solvers require standardized or similarly scaled features. Adding a penalty can require more optimization iterations and increase training time. section 4.3.3
In gradient boosting, L2 regularization smooths learned weights and makes Newton-style optimization steps smaller. XGBoost and LightGBM call the L2 control lambda; alpha denotes L1. Tune these values as hyperparameters: stronger regularization can reduce overfitting, but excessive constraint can contribute to underfitting. section 5.4.2 section 5.4.1
Tune regularization in an XGBoost workflow
For an Airbnb regression task, an Optuna objective can test XGBoost boosters, objectives, and hyperparameters across cross-validation trials while minimizing MAE. The gblinear booster uses a linear base learner with L1 and L2 regularization; the resulting parameter set includes reg_lambda alongside reg_alpha. section 7.2.5
Listing 4.5 L2 regularized linear regression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge, Lasso
column_transform = ColumnTransformer(
[('categories', categorical_onehot_encoding, low_card_categorical),
('numeric', numeric_passthrough, continuous)],
remainder='drop',
verbose_feature_names_out=False,
sparse_threshold=0.0)
polynomial_expansion = PolynomialFeatures(degree=2) #1
model = Ridge(alpha=2500.0) #2
model_pipeline = Pipeline(
[('processing', column_transform),
('polynomial_expansion', polynomial_expansion),
('standardizing', numeric_standardization),
('modeling', model)]
) #3
cv = KFold(5, shuffle=True, random_state=0)
rmse = make_scorer(mean_squared_error, squared=False)
cv_scores = cross_validate(estimator=model_pipeline,
X=data[price_window],
y=target_regression[price_window],
scoring=rmse,
cv=cv,
return_train_score=True,
return_estimator=True) #4
mean_cv = np.mean(cv_scores['test_score'])
std_cv = np.std(cv_scores['test_score'])
fit_time = np.mean(cv_scores['fit_time'])
score_time = np.mean(cv_scores['score_time'])
print(f"{mean_cv:0.3f} ({std_cv:0.3f})",
f"fit: {fit_time:0.2f} secs pred: {score_time:0.2f} secs") #5
Inspect a complete Ridge pipeline that combines preprocessing, polynomial features, standardization, cross-validation, and an L2 strength set through alpha. section 4.3.2
Listing 7.17 Optimizing the XGBoost regressor
import optuna
def objective(trial): #1
params = { #2
'booster': trial.suggest_categorical(
'booster',
['gbtree', 'gblinear']
),
'objective': trial.suggest_categorical(
'objective',
['reg:squarederror', 'reg:gamma', 'reg:tweedie']
),
'n_estimators': trial.suggest_int(
'n_estimators', 100, 1000
),
'learning_rate': trial.suggest_float(
'learning_rate', 0.01, 1.0, log=True
),
'subsample': trial.suggest_float(
'subsample', 0.3, 1.0
),
'colsample_bytree': trial.suggest_float(
'colsample_bytree', 0.3, 1.0
),
'max_depth': trial.suggest_int('max_depth', 1, 7),
'min_child_weight': trial.suggest_int('min_child_weight', 1, 7),
'reg_lambda': trial.suggest_float(
'reg_lambda', 1e-9, 100.0, log=True
),
'reg_alpha': trial.suggest_float(
'reg_alpha', 1e-9, 100.0, log=True
),
}
if params['booster'] == 'gblinear': #3
keys_to_remove = [
"colsample_bytree", "max_depth",
"min_child_weight", "subsample"
]
params = {
key:value for key, value in params.items()
if key not in keys_to_remove
}
if params['objective'] == 'reg:tweedie': #4
# Must be between in range [1, 2) : 1=poisson 2=gamma
params['tweedie_variance_power'] = trial.suggest_float(
'tweedie_variance_power', 1.01, 1.99
)
xgb = XGBRegressor(**params) #5
model_pipeline = Pipeline(
[('processing', column_transform),
('xgb', xgb)]
)
cv_splits = cv.split(X, y=X["neighbourhood_more_than_30"])
cv_scores = cross_validate(
estimator=model_pipeline,
X=X,
y=y,
scoring='neg_mean_absolute_error',
cv=cv_splits
) #6
cv_evaluation = np.mean(
np.abs(cv_scores['test_score'])
) #7
return cv_evaluation #8
sqlite_db = "sqlite:///sqlite.db"
study_name = "optimize_XGBoost_tokyo_airbnb"
study = optuna.create_study(
storage=sqlite_db,
study_name=study_name,
direction="minimize",
load_if_exists=True
) #9
study.optimize(objective, n_trials=100) #10
print(study.best_value) #11
print(study.best_params) #12
Inspect an Optuna objective that searches XGBoost configurations, including regularization-related parameters, against cross-validated MAE. section 7.2.5
