6 Regression Algorithms
This chapter covers
- Introduction to Regression
- Bayesian Linear Regression
- Hierarchical Bayesian Regression
- KNN Regression
- Gaussian Process Regression
In the previous chapter, we looked at supervised algorithms for classification. In this chapter, we focus on supervised learning in which we are trying to predict a continuous quantity. We are going to study four intriguing regression algorithms: Bayesian Linear Regression, Hierarchical Bayesian Regression, KNN regresion and Gaussian Process Regression. We will derive them from first principles. Regression algorithms that predict a continuous quantity find many uses in a variety of applications. For example, predicting the price of financial assets or predicting CO2 levels in the atmosphere. Let’s begin by reviewing the fundamentals of regression.
6.1 Introduction to Regression
In supervised learning, we are given a dataset D={(x1,y1),…,(xn,yn)} consisting of tuples of data x and labels y. The goal of a regression algorithm is to learn a mapping from inputs x to outputs y, where y is a continuous quantity, i.e. y ∈ R.
A regressor f can be viewed as a mapping between a d-dimensional feature vector ϕ(x) and a label y, i.e. f: Rd → R. Regression problems are typically harder (to achieve higher accuracy) compared to classification problems because we are trying to predict a continuous quantity. Moreover, we are often interested in predicting future response variable y based on past training data.