Ordinary least squares
Ordinary least squares is a method for estimating model parameters by minimizing the sum of squared differences between observed values and model predictions.
Geometry for Programmers
Least squares is a technique that chooses model parameters by minimizing the sum of squared discrepancies between predicted and observed values.
- How it is used
- Uses polynomial fitting to turn data points into interpretable curves and trends.
- How deep it goes
- Develops the error metric, generalized polynomial formulation, matrix construction, and SymPy solution, but defers elaborate derivations.
- Examples
- Uses rowing-time data, polynomial degrees from zero upward, plotted trends, and repeated endpoint observations.
- What you can do after
- Build it from scratch with Python and SymPy by assembling and solving the least-squares coefficient system.
Read how Geometry for Programmers explains ordinary least squares →
Effective Data Analysis
An ordinary least squares regression model estimates coefficients by minimizing the sum of squared differences between observed and predicted outcomes.
- How it is used
- Uses linear predictors to estimate outcomes and evaluate model behavior through coefficients, accuracy metrics, and residuals.
- How deep it goes
- Explains fitted equations, coefficient roles, predictive accuracy, residuals, and diagnostic plots without developing estimation internals.
- Examples
- Demonstrates housing-price prediction using square footage, bedrooms, fitted values, residual scatterplots, and residual histograms.
- What you can do after
- Apply it via statsmodels to fit a regression, generate predictions, inspect coefficients, and diagnose residual behavior.
Read how Effective Data Analysis explains ordinary least squares →
Ensemble Methods for Machine Learning
Ordinary least squares is a linear-model fitting method that minimizes the sum of squared residuals.
- How it is used
- Fits linear models for continuous-valued targets using squared loss without regularization.
- How deep it goes
- Defines the unregularized squared-loss objective and contrasts penalties, but does not derive the fitting solution.
- Examples
- Uses a noisy one-feature function, correlated features, scikit-learn estimators, held-out predictions, and MSE or MAD comparisons.
- What you can do after
- Apply OLS via scikit-learn, evaluate held-out predictions, and recognize when regularization may improve robustness.
Read how Ensemble Methods for Machine Learning explains ordinary least squares →
Grokking AI Algorithms, Second Edition
The least-squares method fits a model by minimizing the sum of squared differences between observed and predicted values.
- How it is used
- Fits a straight regression line to data points for estimating a dependent variable from an independent variable.
- How deep it goes
- Explains means, deviations, slope, intercept, and implementation steps, without pursuing theoretical proofs.
- Examples
- Uses carat-and-price data, scatter plots, a subway-route analogy, tabulated calculations, and Python-like pseudocode.
- What you can do after
- Builds a regression line manually or with a short function that returns its intercept, slope, and endpoints.
Read how Grokking AI Algorithms, Second Edition explains ordinary least squares →
Timeless Algorithms, The Foundational Ideas
Ordinary least squares estimates regression parameters by minimizing the sum of squared differences between observed and predicted responses.
- How it is used
- Estimates regression coefficients for describing variation, predicting responses, and assessing predictors.
- How deep it goes
- Derives the squared-residual objective and its equivalence to maximum likelihood under independent normal errors with constant variance.
- Examples
- Uses a fitted regression line and vertical distances from observations to predictions, then models responses around the line with normal errors.
- What you can do after
- Understand why minimizing squared residuals yields the same coefficients as maximum likelihood under normally distributed errors.
Read how Timeless Algorithms, The Foundational Ideas explains ordinary least squares →
Also covered in
- Build AI Drug Discovery PipelinesIn early access (MEAP)
- AI Recommender SystemsIn early access (MEAP)
- Statistics Every Programmer Needs
What to know first
- squared errorSquared error is a measure of discrepancy equal to the square of the difference between a predicted or fitted value and its corresponding observed or target value.
- normal distributionA normal distribution is a continuous probability distribution with a symmetric, bell-shaped density determined by its mean and variance.
- arrayAn array is an ordered collection of elements organized in one or more dimensions and accessible by position or index.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
Related concepts
- maximum likelihood estimationequivalent to · Maximum likelihood estimation is a statistical method that selects parameter values maximizing the likelihood of observed data under a specified probabilistic model.
- squared erroraddresses · Squared error is a measure of discrepancy equal to the square of the difference between a predicted or fitted value and its corresponding observed or target value.
- alternating least squaresused by · Alternating least squares is an iterative optimization algorithm that alternately estimates each factor matrix while holding the others fixed.
- polynomial functionuses · A polynomial function is a function expressible as a finite sum of monomials with constant coefficients and nonnegative integer exponents.
- linear regressionkind of · Linear regression is a statistical method that models a numeric response as a linear function of one or more predictor variables with fitted parameters.
Questions about ordinary least squares
What is ordinary least squares?
Ordinary least squares is a method for estimating model parameters by minimizing the sum of squared differences between observed values and model predictions.
Which Manning books explain ordinary least squares?
Geometry for Programmers, Effective Data Analysis, Ensemble Methods for Machine Learning, Grokking AI Algorithms, Second Edition and Timeless Algorithms, The Foundational Ideas.
What should I know before learning ordinary least squares?
Manning books that teach ordinary least squares usually introduce squared error, normal distribution, array and parameter first.
What is ordinary least squares a kind of?
Ordinary least squares is a kind of linear regression. Linear regression is a statistical method that models a numeric response as a linear function of one or more predictor variables with fitted parameters.
What is ordinary least squares also called?
Ordinary least squares is also called OLS.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




