L1 regularization, also called Lasso regression, adds a penalty equal to the sum of the absolute coefficient values. The penalty constrains a model during optimization and can drive some coefficients exactly to zero. A zero coefficient makes its feature irrelevant to the prediction, so L1 acts as implicit feature selection while reducing model complexity and helping generalization. section 4.3.2 section 4.4
This behavior is especially useful with many candidate predictors. Features less associated with the target tend to receive zero coefficients. When several multicollinear features are similarly predictive, L1 typically retains one nonzero coefficient rather than distributing weight across them. That compactness can simplify a model, but the selected feature set may vary across samples when predictors are highly correlated. section 4.3.2 section 6.2.1
Choose L1 regularization when its trade-offs fit the modeling goal:
- Use it to reduce overfitting by limiting regression-model complexity and removing some features through zero coefficients. section 4.4
- Prefer it over L2 when feature selection is valuable; L2 generally shrinks coefficients instead and is faster to compute. section 4.3.2 section 4.4
- Check stability when predictors are strongly correlated, because different subsamples can produce different nonzero coefficients. section 6.2.1
- Allow sufficient optimization time: penalized logistic regression takes more iterations, with L1 taking longer than L2; limiting max_iter can stop convergence early. section 4.3.3
Lasso in a tabular regression pipeline
A practical workflow preprocesses categorical and numeric columns, creates degree-two polynomial features, standardizes them, and fits sklearn.linear_model.Lasso(alpha=0.1). Five-fold cross-validation evaluates RMSE, while inspecting nonzero coefficients reveals how many transformed features remain active. This contrasts with the adjacent Ridge pipeline, which uses L2 regularization. section 4.3.2
Listing 4.6 L1 regularized linear regression
model = Lasso(alpha=0.1) #1
model_pipeline = Pipeline(
[('processing', column_transform),
('polynomial_expansion', polynomial_expansion),
('standardizing', numeric_standardization),
('modeling', model)]
) #2
cv = KFold(5, shuffle=True, random_state=0)
rmse = make_scorer(mean_squared_error, squared=False)
cv_scores = cross_validate(estimator=model_pipeline,
X=data[price_window],
y=target_regression[price_window],
scoring=rmse,
cv=cv,
return_train_score=True,
return_estimator=True) #3
mean_cv = np.mean(cv_scores['test_score'])
std_cv = np.std(cv_scores['test_score'])
fit_time = np.mean(cv_scores['fit_time'])
score_time = np.mean(cv_scores['score_time'])
print(f"{mean_cv:0.3f} ({std_cv:0.3f})",
f"fit: {fit_time:0.2f} secs pred: {score_time:0.2f} secs") #4
This runnable Lasso pipeline shows where alpha, preprocessing, polynomial expansion, standardization, and cross-validation fit in an end-to-end regression workflow. section 4.3.2
L1 also appears beyond standalone linear regression. In XGBoost and LightGBM, alpha denotes the L1 regularizer and lambda the L2 regularizer; both are tuning parameters intended to reduce overfitting during optimization. An XGBoost gblinear booster uses a linear base learner with L1 and L2 regularization, allowing these settings to be searched alongside booster and objective choices. section 5.4.2 section 7.2.5
