concept ridge regression in category R

appears as: ridge regression, ridge regression
Machine Learning with R, the tidyverse, and mlr

This is an excerpt from Manning's book Machine Learning with R, the tidyverse, and mlr.

By the end of this chapter, I hope you’ll have an intuitive understanding of what regularization is, how it works, and why it’s important. You’ll understand how ridge regression and LASSO work and how they’re useful, and how elastic net is a mixture of them both. Finally, you’ll build ridge regression, LASSO, and elastic net models, and use benchmarking to compare them to each other and to a linear regression model with no regularization.

We then create the loss function for LASSO (the L1-penalized loss function) in exactly the same way we did for ridge regression: we multiply the L1 norm by lambda (which has the same meaning) and add it to the sum of squares. The L1-penalized loss function is shown in equation 11.5. Notice that the only difference between this equation and equation 11.2 is that we take the absolute value of the parameters before summing them, instead of squaring them. Say we had three slopes, one of which was negative: 2.2, –3.1, 0.8. The L1 norm of these three slopes would be 2.2 + 3.1 + 0.8 = 6.1.

equation 11.5.

I can already hear you thinking, “So what? What’s the benefit/difference of using the L1 norm instead of the L2 norm?” Well, ridge regression can shrink parameter estimates toward 0, but they will never actually be 0 (unless the OLS estimate is 0 to begin with). So if you have a machine learning task where you believe all the variables should have some degree of predictive value, ridge regression is great because it won’t remove any variables. But what if you have a large number of variables and/or you want an algorithm that will perform feature selection for you? LASSO is helpful here because unlike ridge regression, LASSO is able to shrink small parameter values to 0, effectively removing that predictor from the model.

Let’s represent this graphically the same way we did for ridge regression. Figure 11.7 shows the contours of the sum of squares for the same two imaginary parameters as in figure 11.6. Instead of forming a circle, the LASSO penalty forms a square, rotated 45o such that its vertices lie along the axes (I guess you could call this a diamond). Can you see that, for the same lambda as in our ridge regression example, the combination of parameters with the smallest sum of squares that touches the diamond is one where parameter β2 is 0? This means the predictor represented by this parameter has been removed from the model.

coefTibInts <- tibble(Coef = rownames(ridgeCoefs),
                  Ridge = as.vector(ridgeCoefs),
                  Lm = as.vector(lmCoefs))
coefUntidyInts <- gather(coefTibInts, key = Model, value = Beta, -Coef)

ggplot(coefUntidyInts, aes(reorder(Coef, Beta), Beta, fill = Model)) +
  geom_bar(stat = "identity", col = "black") +
  facet_wrap(~Model) +
  theme_bw()  +
  theme(legend.position = "none")

# The intercepts are different. The intercept isn't included when
# calculating the L2 norm, but is the value of the outcome when all
# the predictors are zero. Because ridge regression changes the parameter
# estimates of the predictors, the intercept changes as a result.

11.2. What is ridge regression?

In this section, I’ll show you what ridge regression is, how it works, and why it’s useful. Take a look at the example in figure 11.1, which I’ve reproduced from chapter 3. I used this figure in chapter 3 to show you what underfitting and overfitting look like for classification problems. When we underfit the problem, we partition the feature space in a way that doesn’t do a good job of capturing local differences near the decision boundary. When we overfit, we place too much importance on these local differences and end up with a decision boundary that captures much of the noise in the training set, resulting in an overly complex decision boundary.