Elastic net regression fits continuous targets with squared loss while combining L1 and L2 regularization. L1 encourages sparse weights and implicit feature selection; L2 improves robustness when features are correlated. section 7.1.1
Two parameters control the balance: α > 0 sets the tradeoff between prediction loss and total regularization, while ρ, from 0 to 1, sets the L1 share. At ρ = 0, the method becomes ridge regression; at ρ = 1, it becomes LASSO. section 7.1.1
Use elastic net when a linear model needs both feature reduction and stability:
- Prefer it to LASSO when correlated features make choosing only one feature unreliable. section 7.1.1
- Expect some coefficients to become exactly zero, preserving LASSO-like sparsity. section 7.1.1
- Tune α and ρ rather than treating either regularization behavior as fixed. section 7.1.1
A correlated-feature regression
A synthetic target is generated from a noisy linear function, then a second feature is created as a correlated transformation of the first. With α = 0.5 and ρ = 0.5, elastic net assigns a zero weight to one feature while retaining the other, achieving test MSE 2.824 and MAD 1.304. section 7.1.1
Listing 7.1 Linear regression models
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
from sklearn.metrics import mean_squared_error, mean_absolute_error
models = ['OLS Regression', 'Ridge Regression', 'LASSO', 'Elastic Net']
regressors = [LinearRegression(), #1
Ridge(alpha=0.5), #1
Lasso(alpha=0.5), #1
ElasticNet(alpha=0.5, l1_ratio=0.5)] #1
for (model, regressor) in zip(models, regressors):
regressor.fit(Xtrn, ytrn) #2
ypred = regressor.predict(Xtst) #3
mse = mean_squared_error(ytst, ypred) #4
mad = mean_absolute_error(ytst, ypred) #4
print('{0}\'s test set performance: MSE = {1:4.3f}, MAD={2:4.3f}'.
format(model, mse, mad))
print('{0} model: {1} * x + {2}\n'. #5
format(model, regressor.coef_, regressor.intercept_))
Compare ordinary least squares, ridge, LASSO, and elastic net in scikit-learn, then inspect test-set MSE, MAD, and learned coefficients. section 7.1.1
