liveBook · Manning

Ordinary least squares

As explained in Timeless Algorithms, The Foundational Ideas

Ordinary least squares estimates regression parameters by minimizing the sum of squared differences between observed and predicted responses.

Ordinary least squares (OLS) estimates regression coefficients by minimizing the sum of squared residuals. Squaring prevents positive and negative errors from canceling and gives larger errors greater weight. The fitted line captures the overall relationship rather than passing through every observation. section 3.4.1

ε^i=yi-y^i

A residual is the observed response minus its predicted value; OLS minimizes the sum of squared residuals. section 3.4.1

OLS also has a probabilistic interpretation. If errors are independent, normally distributed with mean zero, and have constant variance, maximizing the likelihood of the observations gives the same coefficient estimates as minimizing squared residuals. The geometric and likelihood approaches differ in interpretation, not in their resulting regression line. section 3.4.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Derives the squared-residual objective and its equivalence to maximum likelihood under independent normal errors with constant variance.
Examples
Uses a fitted regression line and vertical distances from observations to predictions, then models responses around the line with normal errors.
What you can do after
Understand why minimizing squared residuals yields the same coefficients as maximum likelihood under normally distributed errors.
How it connects
Connects linear regression, residuals, and maximum likelihood estimation through shared coefficient estimates.

Also explained in

Compare how 5 books explain ordinary least squares →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.