liveBook · Manning

L1 regularization

As explained in Machine Learning System Design

A regularization penalty based on the sum of absolute parameter values that promotes sparse model coefficients.

L1 regularization supports feature selection by shrinking model coefficients. When a feature is not relevant to the target variable, its coefficient can be reduced to zero, effectively removing that feature. section 11.3.3

Use L1 regularization as an embedded feature-selection approach, while keeping its practical tradeoffs in view:

  • Pair it with Lasso regression to select features through an embedded model. section 11.3.3
  • Treat it as a middle ground between filter and wrapper methods in computation and selection quality. section 11.3.3
  • Prefer faster rough cutoffs for initial pruning when many features are obviously useless or hardware is limited. section 11.3.3

Lasso removes features

As the L1 regularization term grows, Lasso regression reduces coefficients to zero one by one, producing a smaller feature subset. section 11.3.3

Figure 11.9 Lasso regression eliminates features one by one by reducing their coefficients to zero as L1 regularization term grows.
figure

Shows how increasing the L1 term eliminates features by driving their coefficients to zero. section 11.3.3

How Machine Learning System Design covers it

How deep it goes
Explains coefficient elimination and method tradeoffs, but does not derive the penalty mathematically.
Examples
Illustrates selection with Lasso regression, where increasing regularization reduces coefficients to zero one by one.
What you can do after
Apply it via Lasso regression to prune features by identifying coefficients reduced to zero.
How it connects
Connects L1 regularization with Lasso regression, feature selection, embedded methods, filter methods, and wrapper methods.

Also explained in

Compare how 8 books explain L1 regularization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.