chapter three

3 The algorithm of estimation: Ronald Fisher’s likelihood principle

 

This chapter covers

  • Ronald Fisher’s On the Mathematical Foundations of Theoretical Statistics (1922) and its role in establishing a rigorous basis for statistical inference
  • Maximum likelihood estimates as a unifying principle for learning parameters directly from data
  • Consistency, efficiency, and sufficiency as the performance standards that distinguish principled estimators from ad hoc methods
  • Fisher’s rejection of prior-based inference—and how likelihood reframed objectivity in statistical reasoning
  • Why Fisher’s likelihood framework still underlies modern regression, machine learning loss functions, and model selection

By the early 20th century, probability and inference had advanced far beyond Bayes’ time, yet foundational problems remained unresolved. Bayes’ Theorem had shown how beliefs could be updated in light of evidence—but it left open other questions: how should unknown parameters be estimated directly from data, without appealing to prior belief at all? What distinguished a sound and repeatable method from an arbitrary one?

3.1 What is likelihood?

3.2 The criteria for a good estimator

3.2.1 Consistency: converging toward truth

3.2.2 Efficiency: getting the most from the data

3.2.3 Sufficiency: capturing all the information

3.2.4 Why likelihood wins: Fisher’s critique of Bayesian inference and the method of moments

3.3 Maximum likelihood as the backbone of modern modeling

3.3.1 From biology and agriculture to modeling: the generalization of maximum likelihood

3.3.2 Core models built on MLE

3.3.3 From likelihood to loss functions

3.3.4 Model selection and evaluation

3.3.5 Fisher Information in practice

3.4 What Fisher set in motion

3.5 Summary