chapter eleven

11 Preventing overfitting: ridge regression, LASSO, and elastic net

 

This chapter covers:

  • What does overfitting look like for regression problems?
  • What is reguchlarization?
  • What are ridge regression, LASSO, and elastic net?
  • What are the L1 and L2 norms and how are they used to shrink parameters?

Our societies are full of checks and balances. In our political systems, parties balance each other to (in theory) find solutions that are at neither extreme of each other’s views. Professional areas, such as financial services, have regulatory bodies that prevent them from doing wrong, and ensure the things they say and do are truthful and correct. When it comes to machine learning, it turns out we can apply our own form of regulation to the learning process to prevent the algorithms from overfitting the training set. We call this regulation in machine learning, regularization.

11.1  What is regularization?

11.2  What is ridge regression?

11.3  What is the L2 norm and how does ridge regression use it?

11.4  What is the L1 norm and how does LASSO use it?

11.5  What is elastic net?

11.6  Building our first ridge, LASSO, and elastic net models

11.6.1  Loading and exploring the Iowa dataset

11.6.2  Training the ridge regression model

11.6.3  Training the LASSO model

11.6.4  Training the elastic net model

11.7  Benchmarking ridge, LASSO, elastic net and OLS against each other

11.8  Strengths and weaknesses of ridge, LASSO and elastic net

11.8.1  Additional exercises

11.9  Summary

11.10  Solutions to exercises