LASSO is a linear regression model that adds an L1 penalty to squared loss. The penalty shrinks weights and can drive some exactly to zero, producing a sparse model that implicitly selects features. section 7.1.1
LASSO is easiest to place beside other linear regression choices:
- OLS uses squared loss without regularization and can become unstable with highly correlated features. section 7.1.1
- Ridge uses an L2 penalty to shrink weights, while LASSO uses an L1 penalty to encourage exact zeros. section 7.1.1
- Elastic net combines L1 and L2 penalties, balancing LASSO-style sparsity with Ridge-style robustness. section 7.1.1
Correlated synthetic features
Create one noisy feature and a second feature roughly three times the first, then split the data into training and test sets. In a scikit-learn comparison, LASSO(alpha=0.5) produced weights [-0.0, -0.79809073], showing implicit feature selection, with test MSE 2.832 and mean absolute deviation 1.304. section 7.1.1
Listing 7.1 Linear regression models
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
from sklearn.metrics import mean_squared_error, mean_absolute_error
models = ['OLS Regression', 'Ridge Regression', 'LASSO', 'Elastic Net']
regressors = [LinearRegression(), #1
Ridge(alpha=0.5), #1
Lasso(alpha=0.5), #1
ElasticNet(alpha=0.5, l1_ratio=0.5)] #1
for (model, regressor) in zip(models, regressors):
regressor.fit(Xtrn, ytrn) #2
ypred = regressor.predict(Xtst) #3
mse = mean_squared_error(ytst, ypred) #4
mad = mean_absolute_error(ytst, ypred) #4
print('{0}\'s test set performance: MSE = {1:4.3f}, MAD={2:4.3f}'.
format(model, mse, mad))
print('{0} model: {1} * x + {2}\n'. #5
format(model, regressor.coef_, regressor.intercept_))
Compare LASSO with OLS, Ridge, and Elastic Net by fitting each model, predicting test labels, and calculating MSE and mean absolute error. section 7.1.1
