Regularization reduces overfitting by limiting model complexity. In regression, it penalizes coefficient magnitudes while fitting predictions, encouraging smaller values and sometimes zero values. This helps a model generalize rather than memorize training patterns, especially when many features make a flexible linear model prone to high variance. section 4.3.2 section 4.4
It is particularly useful when predictors are highly correlated. Multicollinearity can make ordinary linear-regression coefficients unstable or unreliable; shrinking coefficients stabilizes estimates and can concentrate importance on fewer correlated features. Regularization also lets feature engineering add transformations and interactions without leaving complexity unchecked. section 4.3.1 section 4.3.2
The two main coefficient-penalty approaches differ in their practical effect:
- L1 regularization, implemented by Lasso regression, pushes many coefficients to exactly zero. It can make features irrelevant and is useful for feature selection. section 4.4 section 4.3.2
- L2 regularization, implemented by Ridge regression, generally shrinks coefficient sizes without necessarily removing features. It is presented as a faster option for reducing overfitting with many features. section 4.4 section 4.3.2
Polynomial Airbnb price model
A regression pipeline one-hot encodes categories, expands features to degree-two polynomial terms, standardizes them, and fits Ridge with alpha=2500. This pairs richer feature interactions with an L2 penalty, then evaluates predictions through five-fold cross-validation and RMSE. Replacing Ridge with Lasso applies L1 regularization instead. section 4.3.2
Listing 4.5 L2 regularized linear regression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge, Lasso
column_transform = ColumnTransformer(
[('categories', categorical_onehot_encoding, low_card_categorical),
('numeric', numeric_passthrough, continuous)],
remainder='drop',
verbose_feature_names_out=False,
sparse_threshold=0.0)
polynomial_expansion = PolynomialFeatures(degree=2) #1
model = Ridge(alpha=2500.0) #2
model_pipeline = Pipeline(
[('processing', column_transform),
('polynomial_expansion', polynomial_expansion),
('standardizing', numeric_standardization),
('modeling', model)]
) #3
cv = KFold(5, shuffle=True, random_state=0)
rmse = make_scorer(mean_squared_error, squared=False)
cv_scores = cross_validate(estimator=model_pipeline,
X=data[price_window],
y=target_regression[price_window],
scoring=rmse,
cv=cv,
return_train_score=True,
return_estimator=True) #4
mean_cv = np.mean(cv_scores['test_score'])
std_cv = np.std(cv_scores['test_score'])
fit_time = np.mean(cv_scores['fit_time'])
score_time = np.mean(cv_scores['score_time'])
print(f"{mean_cv:0.3f} ({std_cv:0.3f})",
f"fit: {fit_time:0.2f} secs pred: {score_time:0.2f} secs") #5
Use the Ridge pipeline listing to see where polynomial expansion, standardization, coefficient regularization, and cross-validation fit together. section 4.3.2
The same goal appears beyond linear regression. Out-of-core logistic regression can use L1 or L2 regularization when repeated passes over batches risk overfitting. XGBoost incorporates regularization terms in its optimization, and deep-learning models can use dropout layers that randomly set inputs to zero to reduce overfitting. section 4.3.5 section 5.4 section 9.4.2
