How deeply does Ensemble Methods for Machine Learning cover L1 regularization?

Defines the absolute-value penalty and demonstrates its coefficient effect, but does not derive optimization internals.

L1 regularization

As explained in Ensemble Methods for Machine Learning

L1 regularization penalizes the sum of absolute parameter values to constrain a model and encourage sparse parameters.

L1 regularization adds the sum of absolute parameter values to the squared-loss objective. Increasing this penalty shrinks weights and can drive some exactly to zero, producing a sparse model with implicit feature selection. section 7.1.1

A useful diagnostic is a regression problem with highly correlated features. Compared with OLS, LASSO can discard one feature while retaining predictive performance, whereas ridge shrinks correlated weights without necessarily eliminating them.

LASSO on correlated features

Split the two-feature dataset into training and test sets, fit Lasso(alpha=0.5), then evaluate MSE and mean absolute error. The displayed model learns coefficients [-0.0, -0.79809073], setting one coefficient to zero; its test MSE is 2.832 and MAD is 1.304.

How Ensemble Methods for Machine Learning covers it

Cover of Ensemble Methods for Machine Learning

liveBook manning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIL1 regularization adds a penalty proportional to the sum of absolute model coefficients, encouraging sparse coefficient estimates.
liveBook manning.com

Grokking Machine LearningL1 regularization adds the L1 norm of a model's non-bias coefficients as a complexity penalty to its regression error.
liveBook manning.com

Deep Learning with PyTorch, Second EditionA regularization method that adds the sum of absolute weight values to the objective, encouraging sparse parameters.
liveBook manning.com

Machine Learning System DesignA regularization penalty based on the sum of absolute parameter values that promotes sparse model coefficients.
liveBook manning.com

Deep Learning with Python, Third EditionA regularization method that penalizes the absolute values of weights to encourage sparsity.

Compare how 6 books explain L1 regularization →

Related concepts

regularizationkind of · Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
l2 normuses · The L2 norm is the nonnegative Euclidean magnitude of a vector, equal to the square root of the sum of its components’ squares, and to absolute value for a scalar.
feature selectionuses · Feature selection is the process of choosing a subset of input features for use in a modeling, analysis, or prediction task.
ridge regressioncontrasts with · Ridge regression is linear regression that applies an L2 penalty to coefficient magnitudes, shrinking estimates toward zero to reduce overfitting.
lassoused by · LASSO is a linear regression model that uses an L1 penalty to encourage some feature weights to become exactly zero.
XGBClassifiercontrasts with · XGBClassifier is XGBoost’s scikit-learn-compatible estimator for classification using gradient-boosted decision trees.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.