concept L1 norm in category R

appears as: L1 norm, The L1 norm, L1 norms
Machine Learning with R, the tidyverse, and mlr

This is an excerpt from Manning's book Machine Learning with R, the tidyverse, and mlr.

11.4. What is the L1 norm, and how does LASSO use it?

Now that you know about ridge regression, learning how LASSO works will be a simple extension of what you’ve already learned. In this section, I’ll show you what the L1 norm is, how it differs from the L2 norm, and how the least absolute shrinkage and selection operator (LASSO) uses it to shrink parameter estimates.

The L1 norm is only slightly different than the L2 norm. Instead of squaring the parameter values, we take their absolute value instead and then sum them. This is shown in equation 11.4 by the vertical lines around βj.

equation 11.4.

We then create the loss function for LASSO (the L1-penalized loss function) in exactly the same way we did for ridge regression: we multiply the L1 norm by lambda (which has the same meaning) and add it to the sum of squares. The L1-penalized loss function is shown in equation 11.5. Notice that the only difference between this equation and equation 11.2 is that we take the absolute value of the parameters before summing them, instead of squaring them. Say we had three slopes, one of which was negative: 2.2, –3.1, 0.8. The L1 norm of these three slopes would be 2.2 + 3.1 + 0.8 = 6.1.

equation 11.5.

I can already hear you thinking, “So what? What’s the benefit/difference of using the L1 norm instead of the L2 norm?” Well, ridge regression can shrink parameter estimates toward 0, but they will never actually be 0 (unless the OLS estimate is 0 to begin with). So if you have a machine learning task where you believe all the variables should have some degree of predictive value, ridge regression is great because it won’t remove any variables. But what if you have a large number of variables and/or you want an algorithm that will perform feature selection for you? LASSO is helpful here because unlike ridge regression, LASSO is able to shrink small parameter values to 0, effectively removing that predictor from the model.