Ridge regression is linear regression with an L2 penalty: it adds the sum of squared coefficients to the learning objective. Minimizing this penalty shrinks weights toward zero, reducing model complexity while preserving the squared-loss fit. section 7.1.1
The method is especially useful when features are highly correlated. Instead of allowing correlated predictors to produce unstable, individually large coefficients, ridge regression spreads their influence through smaller weights. The parameter α controls the tradeoff: larger regularization favors simpler coefficients, while the squared-loss term prevents the degenerate all-zero model. section 7.1.1
A practical ridge-regression workflow is:
- Create or collect continuous-label data, then split it into training and test sets. section 7.1.1
- Fit Ridge with a chosen α, alongside OLS, LASSO, or elastic net when comparing regularization strategies. section 7.1.1
- Predict on held-out data and compare MSE or MAD; tune α through hyperparameter search when necessary. section 7.1.1 section 7.4.2
Correlated synthetic features
Construct a two-feature dataset in which the second feature is approximately three times the first, add noise, and train competing regressors. OLS learns weights of about [-1.46, -0.32], whereas ridge with α = 0.5 learns approximately [-0.34, -0.70]. Their test errors are similar, but ridge distributes the effect more evenly and is described as more robust to correlated features. section 7.1.1
Listing 7.1 Linear regression models
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
from sklearn.metrics import mean_squared_error, mean_absolute_error
models = ['OLS Regression', 'Ridge Regression', 'LASSO', 'Elastic Net']
regressors = [LinearRegression(), #1
Ridge(alpha=0.5), #1
Lasso(alpha=0.5), #1
ElasticNet(alpha=0.5, l1_ratio=0.5)] #1
for (model, regressor) in zip(models, regressors):
regressor.fit(Xtrn, ytrn) #2
ypred = regressor.predict(Xtst) #3
mse = mean_squared_error(ytst, ypred) #4
mad = mean_absolute_error(ytst, ypred) #4
print('{0}\'s test set performance: MSE = {1:4.3f}, MAD={2:4.3f}'.
format(model, mse, mad))
print('{0} model: {1} * x + {2}\n'. #5
format(model, regressor.coef_, regressor.intercept_))
Use the scikit-learn listing to fit Ridge, compare it with OLS, LASSO, and elastic net, predict on test data, and calculate MSE and MAD. section 7.1.1
