Linear regression predicts a continuous response from a linear combination of features. Given training pairs (xᵢ, yᵢ), let φ(xᵢ) be a feature vector and write yᵢ = wᵀφ(xᵢ) + εᵢ. The coefficients w determine the prediction; εᵢ represents the residual error. Mean squared error measures how far predictions fall from observed responses. section 6.1
A Gaussian error assumption, ε ∼ N(0, σ²), makes fitting the coefficients a likelihood problem. With σ² fixed, minimizing negative log likelihood amounts to minimizing squared prediction errors. A zero-mean Gaussian prior on w adds a penalty for large coefficients, addressing the risk of overfitting. This leads to a regularized linear-regression objective. section 6.2
A from-scratch gradient-descent fit follows the objective directly:
- Add a column of ones to the feature matrix so the weights include an intercept. section 6.2
- Initialize w, compute predictions Xw, and record squared-error loss with the coefficient penalty. section 6.2
- Compute the objective’s gradient, update w using a learning rate, and use the fitted weights to predict on new inputs. section 6.2
California house prices
A regularized fit on the California housing dataset shows loss decreasing over training epochs. In a standardized projection, the fitted trend rises with the average number of rooms. section 6.2
Listing 6.1 Bayesian linear regression
import math
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_california_housing
class ridge_reg():
def __init__(self, n_iter=20, learning_rate=1e-3, lmbda=0.1):
self.n_iter = n_iter
self.learning_rate = learning_rate
self.lmbda = lmbda
def fit(self, X, y):
X = np.insert(X, 0, 1, axis=1) #1
self.loss = []
self.w = np.random.rand(X.shape[1])
for i in range(self.n_iter):
y_pred = X.dot(self.w)
mse = np.mean(0.5*(y - y_pred)**2 +
0.5*self.lmbda*self.w.T.dot(self.w))
self.loss.append(mse)
print(" %d iter, mse: %.4f" %(i, mse))
grad_w = - (y - y_pred).dot(X) + self
.lmbda*self.w #2
self.w -= self.learning_rate * grad_w #3
def predict(self, X):
X = np.insert(X, 0, 1, axis=1) #4
y_pred = X.dot(self.w)
return y_pred
if __name__ == "__main__":
X, y = fetch_california_housing(return_X_y=True)
X_reg = X[:,2].reshape(-1,1) #5
X_std = (X_reg - X_reg.mean())/X.std()
y_std = (y - y.mean())/y.std() #6
X_std = X_std[:200,:]
y_std = y_std[:200]
rr = ridge_reg()
rr.fit(X_std, y_std)
y_pred = rr.predict(X_std)
print(rr.w)
plt.figure()
plt.plot(rr.loss)
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.tight_layout()
plt.show()
plt.figure()
plt.scatter(X_std, y_std)
plt.plot(np.linspace(-1,1), rr.w[1]*np.linspace(-1,1)+rr.w[0], c='red')
plt.xlim([-0.01,0.01])
plt.xlabel("scaled avg num of rooms")
plt.ylabel("scaled house price")
plt.show()
The listing exposes the fit-and-predict implementation, including the intercept column, coefficient penalty, and gradient update. section 6.2
