liveBook · Manning

Chapter 3: The algorithm of estimation: Ronald Fisher and maximum likelihood

What you will learn in chapter 3 of Timeless Algorithms, The Foundational Ideas

The introduction contrasts Fisher’s data-first approach with Bayesian inference and presents maximum likelihood as a way to estimate unknown parameters.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction contrasts Fisher’s data-first approach with Bayesian inference and presents maximum likelihood as a way to estimate unknown parameters.

Key terms: prior beliefs, prior distributions, observed data, likelihood, maximum likelihood estimation, MLE, optimization

3.1 Fisher’s approach to statistical inference

This section contrasts Bayes’s belief-updating approach with Fisher’s use of observed data to compare fixed but unknown parameter values.

Key terms: unknown parameters, prior distribution, posterior distribution, assumed probability model, likelihood

3.1.1 Estimating unknown parameters

The section contrasts Bayesian and Fisherian readings of the same positive test result, then extends Fisher’s comparison of explanations to estimating unknown parameters.

Key terms: prior probability, posterior probability, Bayesian inference, likelihood, likelihood ratio, parameter, estimate, estimator

3.1.2 What is likelihood?

This section distinguishes likelihood from probability by showing how the same statistical model is interpreted differently once the data is observed.

Key terms: probability model, likelihood function, observed data, parameter, likelihood values, relative measure, parameter values

3.1.3 Maximum likelihood estimation

This section defines maximum likelihood estimation as choosing the parameter value that best accounts for observed data, illustrates it with coin tosses, and explains its use across statistical models.

Key terms: maximum likelihood estimation, likelihood function, arg max, theta hat, probability of heads, Bernoulli model, maximum likelihood estimate, parameter, statistical models

3.1.4 Maximum likelihood estimation

This section derives the maximum likelihood estimate of a normal distribution’s unknown mean and shows how the same estimation principle applies across models.

Key terms: normal distribution, independent, likelihood, density, mean 𝜇, variance 𝜎², log-likelihood, natural logarithm, sum of squared deviations, sample mean

3.1.5 Why maximum likelihood endures

This section explains why maximum likelihood endures as a flexible way to estimate parameters by optimization, while noting that an estimate must also be evaluated.

Key terms: maximum likelihood estimation, probability model, likelihood function, parameter values, analytically, numerical algorithms, log-likelihood, loss, estimate, estimator

3.2 The criteria for a good estimator

This section introduces Fisher’s three criteria for judging an estimator beyond merely obtaining an estimate.

Key terms: estimator, consistency, efficiency, sufficiency

3.2.1 Consistency: converging toward truth

This section explains consistency as an estimator’s tendency to approach the true parameter as more data is collected, rather than a promise that each estimate will be accurate.

Key terms: consistent estimator, consistency, sample size, true parameter, sample mean, estimation procedure, statistical model, independent observations, correctly specified probability model, unbiasedness

3.2.2 Efficiency: getting the most from the data

This section explains efficiency as the precision of an unbiased estimator, relates that precision to Fisher Information and the Cramér–Rao lower bound, and shows why it matters for maximum likelihood estimation.

Key terms: efficient estimator, variance, unbiased estimators, precision, Fisher Information, likelihood function, Cramér–Rao lower bound, maximum likelihood estimators, asymptotic efficiency, consistent

3.2.3 Sufficiency: capturing all the information

This section explains sufficiency as model-dependent, lossless reduction of data for estimating a parameter, illustrates it with a sample mean, and places it alongside consistency and efficiency.

Key terms: sufficient statistic, sufficiency, lossless data reduction, statistical model, parameter, sample mean, normal distribution, known variance, population mean, consistency

3.2.4 Fisher’s case for maximum likelihood

This section explains why Fisher regarded maximum likelihood as a principled approach to estimation, contrasts it with Bayesian inference and the method of moments, and notes its continuing influence.

Key terms: consistency, efficiency, sufficiency, maximum likelihood estimation, regularity conditions, Bayesian inference, prior distributions, fixed but unknown constants, likelihood function, method of moments

3.3 Maximum likelihood and logistic regression

The section explains maximum likelihood as a model-independent way to estimate parameters and introduces logistic regression as a practical illustration.

Key terms: maximum likelihood estimation, probability model, likelihood function, parameters, logistic regression, binary outcomes, predictor variables, estimated probability, decision threshold

3.3.1 Why logistic regression requires maximum likelihood

This section explains why logistic regression’s coefficients cannot be solved for directly and how maximum likelihood estimates them instead.

Key terms: regression coefficients, predictor variables, binary outcomes, closed-form solution, maximum likelihood estimation, likelihood function, numerical optimization

3.3.2 The logistic model

This section explains how logistic regression turns predictors into binary-outcome probabilities, learns its coefficients by maximum likelihood, and uses the resulting probabilities for classification.

Key terms: linear predictor, log-odds, intercept, regression coefficients, logistic function, sigmoid function, predicted probability, Bernoulli random variable, likelihood function, independent observations

3.4 Linear regression as a special case

This section introduces linear regression and explains why ordinary least squares is a special case of maximum likelihood estimation when model errors are normally distributed.

Key terms: continuous response variable, explanatory variables, simple linear regression, multiple linear regression, ordinary least squares (OLS), sum of the squared prediction errors, normal distribution, maximum likelihood estimator

3.4.1 Primer on linear regression

This section introduces simple linear regression, explains how to interpret its predictions and residuals, and shows how ordinary least squares fits a line.

Key terms: continuous response variable, explanatory variable, intercept, slope, error term, predicted value, residual, prediction error, ordinary least squares, OLS

3.4.2 A maximum likelihood interpretation

This section shows why maximum likelihood and ordinary least squares produce the same linear regression coefficients under independent, normally distributed errors.

Key terms: model errors, normally distributed, expected value, error variance, likelihood, log-likelihood, sum of the squared residuals, ordinary least squares, maximum likelihood

3.4.3 Worked example

A study-hours example shows how Gaussian maximum likelihood fits a regression line, interprets its predictions and residuals, and relates the fit to ordinary least squares.

Key terms: maximum likelihood, log-likelihood, score functions, 𝛽0, 𝛽1, slope, intercept, prediction, residual, error variance

3.4.4 Why linear regression mattered

This section explains why linear regression helped establish maximum likelihood as a unifying framework and how the same principle appears in modern model training.

Key terms: linear regression, maximum likelihood, normally distributed errors, loss functions, objective functions, likelihood, model parameters

3.5 Maximum likelihood and neural networks

This section connects neural-network training to maximum likelihood, showing how common losses express likelihood objectives even though neural networks require different optimization methods.

Key terms: maximum likelihood, loss function, negative log-likelihood, closed-form solutions, gradient descent, network weights, cross-entropy loss, categorical probability model, mean squared error, normally distributed errors

3.6 MLE beyond regression analysis and neural networks

This section explains that maximum likelihood estimation extends beyond regression and neural networks because the same framework can be applied to many kinds of data once a probability model is specified.

Key terms: maximum likelihood estimation, probability model, likelihood principle

3.6.1 Poisson and exponential models

This section distinguishes Poisson counts from exponential waiting times and shows how maximum likelihood estimates the event rate in each model.

Key terms: Poisson distribution, exponential distribution, fixed interval, time between successive events, maximum likelihood, event rate, sample mean, observed counts, observed waiting times, closed-form estimators

3.6.2 Naïve Bayes

The section shows how Naïve Bayes combines maximum-likelihood parameter estimation with Bayesian classification.

Key terms: Naive Bayes, maximum likelihood, conditionally independent, joint likelihood, posterior probability

3.6.3 Hidden Markov Models

This section explains how maximum likelihood estimation applies to sequential data with hidden states, using an iterative method rather than a closed-form solution.

Key terms: Hidden Markov Models, hidden variables, Baum-Welch algorithm, Expectation-Maximization (EM) algorithm, likelihood

3.7 Model Selection

Introduces model selection as choosing among models fitted to the same data while balancing fit against complexity.

Key terms: model selection, parameter estimation, goodness of fit, model complexity, maximum likelihood

3.7.1 Likelihood Ratio Tests

This section explains how a likelihood ratio test compares nested models and judges whether added parameters improve the fit enough to keep them.

Key terms: likelihood ratio test, nested models, added parameters, maximized likelihoods, maximum likelihood estimates, test statistic, approximate chi-square distribution, statistical significance

3.7.2 Akaike and Bayesian Information Criteria

This section explains how AIC and BIC use maximized likelihood and a complexity penalty to compare competing models, including models that are not nested.

Key terms: Akaike Information Criterion, Bayesian Information Criterion, maximized likelihood, model complexity, parsimony, AIC, BIC, predictive performance, sample size

3.8 Fisher's enduring legacy

This section explains why Fisher’s maximum likelihood principle endures and how its approach to inference complements Bayesian reasoning.

Key terms: maximum likelihood estimation, probability model, likelihood, observed data, Bayesian inference, prior beliefs, statistical inference

3.9 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 2: From effect to cause: Thomas Bayes and the first algorithm of learning Chapter 4: From evidence to decisions: Neyman, Pearson, and the principles of hypothesis testing →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.