liveBook · Manning

Ordinary least squares

As explained in Ensemble Methods for Machine Learning

Ordinary least squares is a linear-model fitting method that minimizes the sum of squared residuals.

Ordinary least squares (OLS) fits a linear model by minimizing squared prediction errors. It treats model fitting as an unregularized learning problem: squared loss controls fit, while no penalty controls complexity. section 7.1.1

OLS is useful for continuous-valued targets, but its unregularized coefficients can become unstable when features are highly correlated. Ridge, LASSO, and elastic net retain squared loss while adding different penalties. section 7.1.1 section 7.5

Noisy synthetic regression

Generate 100 one-feature examples from a noisy version of f(x) = -2.5x + 3.2, fit OLS alongside Ridge, LASSO, and elastic net, then compare predictions on held-out data. With two correlated features, the OLS coefficients become unstable relative to regularized alternatives.

How Ensemble Methods for Machine Learning covers it

How deep it goes
Defines the unregularized squared-loss objective and contrasts penalties, but does not derive the fitting solution.
Examples
Uses a noisy one-feature function, correlated features, scikit-learn estimators, held-out predictions, and MSE or MAD comparisons.
What it assumes
Assumes familiarity with machine-learning loss and regularization while reinforcing their roles in linear-model fitting.
What you can do after
Apply OLS via scikit-learn, evaluate held-out predictions, and recognize when regularization may improve robustness.
How it connects
Connects OLS with linear regression, squared loss, multicollinearity, Ridge, LASSO, elastic net, and generalized linear models.
Cover of Ensemble Methods for Machine Learning
liveBookmanning.com
Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

Compare how 4 books explain ordinary least squares →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.