Chapter 3: The algorithm of estimation: Ronald Fisher and maximum likelihood
The introduction contrasts Fisher’s data-first approach with Bayesian inference and presents maximum likelihood as a way to estimate unknown parameters.
After this chapter you will be able to
- Maximum Likelihood Principle. Explain that Fisher’s likelihood principle favors the explanation assigning the highest likelihood to the observed data, and that MLE uses this idea to estimate unknown values by fitting a model through optimization.
- Bayesian and Fisherian inference. Distinguish how Bayes and Fisher use observed data to learn about unknown parameters. Explain that Fisher treats parameters as fixed but unknown and compares how plausible competing values make the observations under an assumed probability model, rather than assigning probabilities to the parameters.
- Fisherian likelihood comparison. Describe how Fisher treats the positive result as fixed and compares how well disease and no disease explain it. Distinguish the resulting 99-to-1 likelihood comparison from a probability that the patient has the disease.
- Extension from explanations to parameters. Explain how comparing two disease explanations extends to comparing possible values of an unknown parameter, and distinguish a parameter, an estimate, and an estimator.
- Fixed Data Versus Fixed Parameters. Distinguish a probability model from a likelihood function by stating what is held fixed and what varies in each. Apply the distinction to ten coin tosses with seven observed heads.
- Relative Meaning of Likelihood. Explain what likelihood values tell us about competing parameter values and why they are not probabilities assigned to those parameters.
- Maximum likelihood selection rule. Explain that maximum likelihood estimation selects the parameter value that maximizes the likelihood of the observed data, rather than averaging parameter values or incorporating prior beliefs. Describe how arg max identifies the parameter value, not the maximum likelihood number.
- Log-Likelihood Mean Derivation. Explain why taking the logarithm simplifies maximization and trace how the normal-model estimate of the mean becomes the sample mean.
- General Maximum Likelihood Principle. State the procedure that carries maximum likelihood estimation beyond the normal example, and describe how the resulting estimator can depend on the model.
- General framework for estimation. Explain how maximum likelihood turns parameter estimation into a reusable optimization problem: specify a probability model, construct its likelihood function, and choose the parameter values under which the observed data is most plausible.
- Three criteria for estimators. Explain why obtaining a parameter estimate is only a first step, and describe consistency, efficiency, and sufficiency as Fisher’s criteria for evaluating an estimator.
- Long-run convergence of estimates. Explain that a consistent estimator becomes increasingly likely to approach the true parameter as sample size grows, even though individual estimates may be inaccurate or fluctuate.
- Efficiency as estimator precision. Explain how efficiency distinguishes estimators that may both converge toward the true parameter: the more efficient one produces less variable estimates, subject to the statistical model.
- Information and precision limit. Describe how changes in likelihood indicate Fisher Information and how the Cramér–Rao lower bound limits the variance of unbiased estimators.
- Model-dependent sufficient statistics. Explain that a statistic is sufficient when it retains all the observed data’s information relevant to a particular parameter, so returning to the original observations cannot improve inference about it. State that sufficiency depends on the statistical model.
- Fisher’s case for likelihood. Explain how Fisher used consistency, efficiency, and sufficiency to evaluate estimators, and why he favored maximum likelihood under appropriate regularity conditions.
- Maximum likelihood estimation principle. Explain why maximum likelihood estimation is not a statistical model: after a probability model is specified, it selects the parameter values under which the observed data are most plausible.
- Logistic coefficients and direct solutions. Explain what logistic regression estimates and why its coefficients cannot be obtained through a closed-form solution.
- Maximum likelihood coefficient estimation. Describe how maximum likelihood selects logistic regression coefficients and why finding them requires numerical optimization.
- Linear predictor and log-odds. Explain that logistic regression combines predictor values and coefficients into a linear predictor representing the log-odds of the positive outcome, not the predicted outcome itself. Describe how the intercept and coefficient signs affect those log-odds.
- Sigmoid probability transformation. Describe how the sigmoid function converts the unbounded linear predictor into a probability of the positive class, including what happens around zero and toward either extreme.
- Bernoulli likelihood for binary outcomes. Explain how each observed binary outcome contributes its predicted probability to a likelihood and how independent observations combine into a likelihood for the data set.
- Log-likelihood coefficient estimation. Explain why logistic regression maximizes the log-likelihood and why finding its coefficients requires iterative numerical optimization rather than a closed-form solution.
- Least squares likelihood equivalence. Explain what ordinary least squares minimizes and why its solution is also a maximum likelihood estimate under the stated error assumption.
- Simple linear regression model. Explain what linear regression relates and how to interpret the parts of a simple linear regression model.
- Ordinary least squares fitting. Explain how ordinary least squares chooses a regression line and why it squares residuals.
- Likelihood and least squares equivalence. Explain why maximizing likelihood gives the same regression coefficients as ordinary least squares under the stated assumptions.
- Gaussian maximum likelihood fit. Explain how the study-hours regression example selects its slope and intercept by maximum likelihood.
- MLE and OLS equivalence. Explain why maximum likelihood and ordinary least squares yield the same fitted regression line in this example.
- Regression and likelihood equivalence. Explain why linear regression mattered to Fisher’s maximum likelihood framework: with normally distributed errors, geometric optimization and probabilistic estimation produce the same solution.
- Loss functions and likelihood. Explain how minimizing a loss function can, for many modern models, be mathematically equivalent to maximizing the probability of observed data under an assumed statistical model.
- Loss and likelihood equivalence. Explain why minimizing many common neural-network losses is equivalent to maximizing the likelihood of the observed data under an assumed probability model.
- Breadth of likelihood estimation. Explain why maximum likelihood estimation is not limited to regression or neural networks: once a probability model is specified, its general estimation framework can apply to many forms of data.
- Maximum likelihood rate estimates. State how maximum likelihood estimates the event rate for Poisson counts and for exponential waiting times, and explain what the different formulas illustrate.
- Naïve Bayes estimation and inference. Explain how Naïve Bayes uses maximum likelihood to estimate probabilities from training data, then uses Bayes’ Theorem to calculate class posterior probabilities. Describe how conditional independence makes the joint likelihood a product of individual feature likelihoods.
- Hidden-state likelihood estimation. Explain why hidden states make parameter estimation in Hidden Markov Models challenging and how Baum-Welch iteratively increases the likelihood of observed sequences. Describe how this differs computationally from regression while retaining the same likelihood-based objective.
- Model selection and complexity. Distinguish model selection from parameter estimation, and explain why choosing among models fitted to the same data requires balancing goodness of fit against complexity. State that maximum likelihood provides a foundation for this comparison, without describing the tools' mechanics.
- Likelihood Improvement and Significance. Explain how the models’ maximized likelihoods inform the preference for the simpler or more complex model, and how statistical significance is assessed.
- Likelihood and complexity tradeoff. Explain why information criteria offer an alternative when competing models are not nested, and how AIC and BIC balance a fitted model’s maximized likelihood against its complexity.
- Enduring maximum likelihood principle. Explain that, given a probability model, maximum likelihood estimation chooses the parameter values that make the observed data most plausible, and that this same objective applies across different learning methods.
What it builds on
- From prior to posterior: how Bayes’ Theorem works (chapter 2) · Bayesian updating from prior beliefs to posterior probabilities.
- Bayesian inference in real-world scenarios (chapter 2) · The rare-disease example and Bayesian updating
- Expressing likelihoods: odds and expected value (chapter 2) · The distinction between probability and odds.
- Linking outcomes: the logic of the multiplication rule (chapter 2) · Multiplying probabilities under an independence assumption.
- Bayesian inference in modern algorithms (chapter 2) · Bayesian inference in classification
Section by section
Introduction
The introduction contrasts Fisher’s data-first approach with Bayesian inference and presents maximum likelihood as a way to estimate unknown parameters.
- Fisher’s Data-First Inference: Distinguish Fisher’s starting point for inference from Bayes’s: Fisher began with observed data rather than prior beliefs, which he considered potentially subjective or arbitrary.
Why it matters: This contrast explains why Fisher sought a data-based foundation for estimation. - Maximum Likelihood Principle: Explain that Fisher’s likelihood principle favors the explanation assigning the highest likelihood to the observed data, and that MLE uses this idea to estimate unknown values by fitting a model through optimization.
Why it matters: This is the chapter’s central idea for estimating unknown parameters from data.
Key terms: prior beliefs, prior distributions, observed data, likelihood, maximum likelihood estimation, MLE, optimization
3.1 Fisher’s approach to statistical inference
This section contrasts Bayes’s belief-updating approach with Fisher’s use of observed data to compare fixed but unknown parameter values.
- Bayesian and Fisherian inference: Distinguish how Bayes and Fisher use observed data to learn about unknown parameters. Explain that Fisher treats parameters as fixed but unknown and compares how plausible competing values make the observations under an assumed probability model, rather than assigning probabilities to the parameters.
Why it matters: This contrast explains what Fisher’s likelihood-based approach is trying to accomplish without treating parameters as random quantities.
Key terms: unknown parameters, prior distribution, posterior distribution, assumed probability model, likelihood
3.1.1 Estimating unknown parameters
The section contrasts Bayesian and Fisherian readings of the same positive test result, then extends Fisher’s comparison of explanations to estimating unknown parameters.
- Bayesian interpretation of a positive test: Explain why a positive result raises the probability of this rare disease from 0.1% to approximately 9%, rather than making disease the most likely explanation.
Why it matters: This result makes clear why strong test evidence is not, by itself, a high probability of disease. - Fisherian likelihood comparison: Describe how Fisher treats the positive result as fixed and compares how well disease and no disease explain it. Distinguish the resulting 99-to-1 likelihood comparison from a probability that the patient has the disease.
Why it matters: A likelihood ratio and a posterior probability answer different questions, even when they use the same test result. - Extension from explanations to parameters: Explain how comparing two disease explanations extends to comparing possible values of an unknown parameter, and distinguish a parameter, an estimate, and an estimator.
Why it matters: This shift connects the diagnostic example to estimation problems involving many possible numerical values.
Key terms: prior probability, posterior probability, Bayesian inference, likelihood, likelihood ratio, parameter, estimate, estimator
3.1.2 What is likelihood?
This section distinguishes likelihood from probability by showing how the same statistical model is interpreted differently once the data is observed.
- Fixed Data Versus Fixed Parameters: Distinguish a probability model from a likelihood function by stating what is held fixed and what varies in each. Apply the distinction to ten coin tosses with seven observed heads.
Why it matters: This change in perspective turns a model of possible outcomes into a way to compare explanations of observed data. - Relative Meaning of Likelihood: Explain what likelihood values tell us about competing parameter values and why they are not probabilities assigned to those parameters.
Why it matters: It prevents a comparison of parameter values from being mistaken for a statement of their probabilities.
Key terms: probability model, likelihood function, observed data, parameter, likelihood values, relative measure, parameter values
3.1.3 Maximum likelihood estimation
This section defines maximum likelihood estimation as choosing the parameter value that best accounts for observed data, illustrates it with coin tosses, and explains its use across statistical models.
- Maximum likelihood selection rule: Explain that maximum likelihood estimation selects the parameter value that maximizes the likelihood of the observed data, rather than averaging parameter values or incorporating prior beliefs. Describe how arg max identifies the parameter value, not the maximum likelihood number.
Why it matters: This selection rule is the basis for the estimation examples that follow. - Coin toss likelihood estimate: Apply the selection rule to ten coin tosses with seven heads: state that the estimated probability of heads is 0.7 because it makes the observed outcome most plausible under the Bernoulli model.
Why it matters: The coin example makes the selection rule concrete. - Generality across statistical models: Explain that the same likelihood-maximizing objective applies to different kinds of parameters and is not tied to one distribution or application.
Why it matters: The coin example illustrates a principle that also applies to much more complex models.
Key terms: maximum likelihood estimation, likelihood function, arg max, theta hat, probability of heads, Bernoulli model, maximum likelihood estimate, parameter, statistical models
3.1.4 Maximum likelihood estimation
This section derives the maximum likelihood estimate of a normal distribution’s unknown mean and shows how the same estimation principle applies across models.
- Independent Sample Likelihood: Describe how the likelihood is constructed for a random sample from a normal distribution with unknown mean and known variance.
Why it matters: Constructing the sample likelihood gives the estimation problem something to maximize. - Log-Likelihood Mean Derivation: Explain why taking the logarithm simplifies maximization and trace how the normal-model estimate of the mean becomes the sample mean.
Why it matters: The familiar sample mean follows from optimizing an assumed probability model, rather than merely choosing an intuitive measure of center. - General Maximum Likelihood Principle: State the procedure that carries maximum likelihood estimation beyond the normal example, and describe how the resulting estimator can depend on the model.
Why it matters: The reusable principle, not the particular sample-mean result, is what makes maximum likelihood useful across statistical models and machine learning.
Key terms: normal distribution, independent, likelihood, density, mean 𝜇, variance 𝜎², log-likelihood, natural logarithm, sum of squared deviations, sample mean
3.1.5 Why maximum likelihood endures
This section explains why maximum likelihood endures as a flexible way to estimate parameters by optimization, while noting that an estimate must also be evaluated.
- General framework for estimation: Explain how maximum likelihood turns parameter estimation into a reusable optimization problem: specify a probability model, construct its likelihood function, and choose the parameter values under which the observed data is most plausible.
Why it matters: This shared principle explains why maximum likelihood is useful across otherwise different statistical models. - Analytical and numerical optimization: Distinguish finding a likelihood maximum analytically in simple cases from using numerical algorithms in complex cases, and describe the connection to minimizing a loss in machine learning.
Why it matters: It connects statistical estimation to the optimization used to fit many machine-learning models. - Estimation versus estimator evaluation: Explain why obtaining an estimate is not the whole task: an estimator must also be evaluated for how reliably it recovers the unknown parameter across repeated cycles.
Why it matters: It separates the ability to produce an estimate from the question of whether that estimate is reliable.
Key terms: maximum likelihood estimation, probability model, likelihood function, parameter values, analytically, numerical algorithms, log-likelihood, loss, estimate, estimator
3.2 The criteria for a good estimator
This section introduces Fisher’s three criteria for judging an estimator beyond merely obtaining an estimate.
- Three criteria for estimators: Explain why obtaining a parameter estimate is only a first step, and describe consistency, efficiency, and sufficiency as Fisher’s criteria for evaluating an estimator.
Why it matters: These criteria provide a way to judge the quality of an estimate, not just calculate one.
Key terms: estimator, consistency, efficiency, sufficiency
3.2.1 Consistency: converging toward truth
This section explains consistency as an estimator’s tendency to approach the true parameter as more data is collected, rather than a promise that each estimate will be accurate.
- Long-run convergence of estimates: Explain that a consistent estimator becomes increasingly likely to approach the true parameter as sample size grows, even though individual estimates may be inaccurate or fluctuate.
Why it matters: Consistency explains why collecting more evidence can make an estimation procedure more reliable. - Conditions and scope of consistency: Distinguish consistency as a property of an estimation procedure, conditional on model assumptions, from unbiasedness, which concerns average behavior in finite samples.
Why it matters: The distinction prevents a reliable long-run tendency from being mistaken for a guarantee about one estimate or a finite sample.
Key terms: consistent estimator, consistency, sample size, true parameter, sample mean, estimation procedure, statistical model, independent observations, correctly specified probability model, unbiasedness
3.2.2 Efficiency: getting the most from the data
This section explains efficiency as the precision of an unbiased estimator, relates that precision to Fisher Information and the Cramér–Rao lower bound, and shows why it matters for maximum likelihood estimation.
- Efficiency as estimator precision: Explain how efficiency distinguishes estimators that may both converge toward the true parameter: the more efficient one produces less variable estimates, subject to the statistical model.
Why it matters: Efficiency lets you compare how precisely estimators use the same available evidence. - Information and precision limit: Describe how changes in likelihood indicate Fisher Information and how the Cramér–Rao lower bound limits the variance of unbiased estimators.
Why it matters: These ideas make precision a limit that can be reasoned about, rather than just a visual comparison of estimates. - Asymptotic efficiency of MLE: Explain why maximum likelihood estimators can be attractive on efficiency grounds as sample size grows, without claiming that every MLE is optimal in every sample.
Why it matters: It explains why Fisher saw maximum likelihood as more than a convenient way to calculate an estimate.
Key terms: efficient estimator, variance, unbiased estimators, precision, Fisher Information, likelihood function, Cramér–Rao lower bound, maximum likelihood estimators, asymptotic efficiency, consistent
3.2.3 Sufficiency: capturing all the information
This section explains sufficiency as model-dependent, lossless reduction of data for estimating a parameter, illustrates it with a sample mean, and places it alongside consistency and efficiency.
- Model-dependent sufficient statistics: Explain that a statistic is sufficient when it retains all the observed data’s information relevant to a particular parameter, so returning to the original observations cannot improve inference about it. State that sufficiency depends on the statistical model.
Why it matters: It explains when a smaller summary can replace the full data set without weakening estimation. - Normal-mean sufficiency example: Describe why the sample mean of 5, 7, 6, 8, and 4, which is 6, is sufficient for the population mean under the stated normal model with known variance.
Why it matters: The example makes clear that sufficiency is a claim about a particular parameter under a particular model. - Sufficiency among estimator criteria: Distinguish sufficiency from consistency and efficiency as criteria for a good estimator: it avoids discarding parameter-relevant information, while the other two concern convergence toward the correct value and making the best use of available information.
Why it matters: It keeps three related standards for evaluating estimators from being mistaken for the same property.
Key terms: sufficient statistic, sufficiency, lossless data reduction, statistical model, parameter, sample mean, normal distribution, known variance, population mean, consistency
3.2.4 Fisher’s case for maximum likelihood
This section explains why Fisher regarded maximum likelihood as a principled approach to estimation, contrasts it with Bayesian inference and the method of moments, and notes its continuing influence.
- Fisher’s case for likelihood: Explain how Fisher used consistency, efficiency, and sufficiency to evaluate estimators, and why he favored maximum likelihood under appropriate regularity conditions.
Why it matters: This connects the way maximum likelihood chooses an estimate to Fisher’s standards for judging its quality. - Fisher’s Bayesian objection: Describe Fisher’s objection to Bayesian priors and distinguish his position from the section’s account of Bayesian inference today.
Why it matters: It separates Fisher’s historical argument from a claim that Bayesian inference has been displaced. - Method of moments comparison: Describe how the method of moments estimates parameters and why Fisher considered it weaker than maximum likelihood.
Why it matters: The comparison makes Fisher’s emphasis on using the model’s information concrete. - Enduring role of maximum likelihood: State the continuing role the section attributes to maximum likelihood in statistics and machine learning.
Why it matters: Its reach explains why Fisher’s estimation principle matters beyond its original statistical setting.
Key terms: consistency, efficiency, sufficiency, maximum likelihood estimation, regularity conditions, Bayesian inference, prior distributions, fixed but unknown constants, likelihood function, method of moments
3.3 Maximum likelihood and logistic regression
The section explains maximum likelihood as a model-independent way to estimate parameters and introduces logistic regression as a practical illustration.
- Maximum likelihood estimation principle: Explain why maximum likelihood estimation is not a statistical model: after a probability model is specified, it selects the parameter values under which the observed data are most plausible.
Why it matters: This distinction helps you recognize the same estimation principle across different models. - Logistic regression predictions: Describe logistic regression as a model for binary outcomes that uses predictor variables to estimate an outcome probability, then compares that probability with a predetermined threshold to make a prediction.
Why it matters: Logistic regression makes the general estimation principle concrete in a familiar prediction task.
Key terms: maximum likelihood estimation, probability model, likelihood function, parameters, logistic regression, binary outcomes, predictor variables, estimated probability, decision threshold
3.3.1 Why logistic regression requires maximum likelihood
This section explains why logistic regression’s coefficients cannot be solved for directly and how maximum likelihood estimates them instead.
- Logistic coefficients and direct solutions: Explain what logistic regression estimates and why its coefficients cannot be obtained through a closed-form solution.
Why it matters: This explains why specifying a logistic model does not by itself give you fitted coefficients. - Maximum likelihood coefficient estimation: Describe how maximum likelihood selects logistic regression coefficients and why finding them requires numerical optimization.
Why it matters: It connects the absence of a direct solution to the method used to fit logistic regression.
Key terms: regression coefficients, predictor variables, binary outcomes, closed-form solution, maximum likelihood estimation, likelihood function, numerical optimization
3.3.2 The logistic model
This section explains how logistic regression turns predictors into binary-outcome probabilities, learns its coefficients by maximum likelihood, and uses the resulting probabilities for classification.
- Linear predictor and log-odds: Explain that logistic regression combines predictor values and coefficients into a linear predictor representing the log-odds of the positive outcome, not the predicted outcome itself. Describe how the intercept and coefficient signs affect those log-odds.
Why it matters: Interpreting the linear predictor correctly prevents treating a log-odds score as a probability. - Sigmoid probability transformation: Describe how the sigmoid function converts the unbounded linear predictor into a probability of the positive class, including what happens around zero and toward either extreme.
Why it matters: The sigmoid makes a linear log-odds score usable as a valid, interpretable probability. - Bernoulli likelihood for binary outcomes: Explain how each observed binary outcome contributes its predicted probability to a likelihood and how independent observations combine into a likelihood for the data set.
Why it matters: The likelihood connects the model’s probabilities to the outcomes actually observed. - Log-likelihood coefficient estimation: Explain why logistic regression maximizes the log-likelihood and why finding its coefficients requires iterative numerical optimization rather than a closed-form solution.
Why it matters: Estimation is what turns the logistic model’s unspecified coefficients into a fitted model that can predict. - Probabilities and class decisions: Trace how fitted coefficients produce probabilities and how a separate decision threshold converts those probabilities into class labels, using the student study-hours example.
Why it matters: Keeping probabilities distinct from class labels preserves the uncertainty the fitted model expresses.
Key terms: linear predictor, log-odds, intercept, regression coefficients, logistic function, sigmoid function, predicted probability, Bernoulli random variable, likelihood function, independent observations
3.4 Linear regression as a special case
This section introduces linear regression and explains why ordinary least squares is a special case of maximum likelihood estimation when model errors are normally distributed.
- Linear regression models and uses: Describe what linear regression relates, what it is used for, and how simple and multiple linear regression differ.
Why it matters: Identifying the response and predictors makes the estimation discussion concrete. - Least squares likelihood equivalence: Explain what ordinary least squares minimizes and why its solution is also a maximum likelihood estimate under the stated error assumption.
Why it matters: The normal-error assumption connects a familiar way of fitting a line to Fisher’s likelihood framework.
Key terms: continuous response variable, explanatory variables, simple linear regression, multiple linear regression, ordinary least squares (OLS), sum of the squared prediction errors, normal distribution, maximum likelihood estimator
3.4.1 Primer on linear regression
This section introduces simple linear regression, explains how to interpret its predictions and residuals, and shows how ordinary least squares fits a line.
- Simple linear regression model: Explain what linear regression relates and how to interpret the parts of a simple linear regression model.
Why it matters: Interpreting the model makes its fitted coefficients and predictions meaningful. - Predictions and residuals: Describe what a fitted regression model predicts and what a residual says about its fit to an observation.
Why it matters: Residuals show how closely the fitted line matches observations without demanding a perfect match to every point. - Ordinary least squares fitting: Explain how ordinary least squares chooses a regression line and why it squares residuals.
Why it matters: This explains the fitting rule behind the regression line, rather than treating the line as an arbitrary visual choice.
Key terms: continuous response variable, explanatory variable, intercept, slope, error term, predicted value, residual, prediction error, ordinary least squares, OLS
3.4.2 A maximum likelihood interpretation
This section shows why maximum likelihood and ordinary least squares produce the same linear regression coefficients under independent, normally distributed errors.
- Normal error model: Describe the assumptions about linear regression errors and what they imply for responses around the regression line.
Why it matters: These assumptions give the regression line a probabilistic interpretation. - Likelihood and least squares equivalence: Explain why maximizing likelihood gives the same regression coefficients as ordinary least squares under the stated assumptions.
Why it matters: It shows how a geometric error-minimization method can also be understood as probabilistic estimation.
Key terms: model errors, normally distributed, expected value, error variance, likelihood, log-likelihood, sum of the squared residuals, ordinary least squares, maximum likelihood
3.4.3 Worked example
A study-hours example shows how Gaussian maximum likelihood fits a regression line, interprets its predictions and residuals, and relates the fit to ordinary least squares.
- Gaussian maximum likelihood fit: Explain how the study-hours regression example selects its slope and intercept by maximum likelihood.
Why it matters: It connects the probability model to the fitted regression coefficients. - Fitted coefficients and residual: Interpret the fitted slope and intercept, and describe what a positive residual means for the seven-hour observation.
Why it matters: It turns the fitted line into an interpretation of scores and individual prediction errors. - Estimated error spread: Describe what the maximum likelihood estimate of error variance represents and how its calculation differs from the variance estimate commonly reported with ordinary least squares.
Why it matters: The fitted line alone does not describe how widely scores vary around it. - MLE and OLS equivalence: Explain why maximum likelihood and ordinary least squares yield the same fitted regression line in this example.
Why it matters: It shows why two different fitting viewpoints agree on the line without requiring identical variance estimates.
Key terms: maximum likelihood, log-likelihood, score functions, 𝛽0, 𝛽1, slope, intercept, prediction, residual, error variance
3.4.4 Why linear regression mattered
This section explains why linear regression helped establish maximum likelihood as a unifying framework and how the same principle appears in modern model training.
- Regression and likelihood equivalence: Explain why linear regression mattered to Fisher’s maximum likelihood framework: with normally distributed errors, geometric optimization and probabilistic estimation produce the same solution.
Why it matters: It shows why maximum likelihood is more than an alternative way to estimate regression parameters. - Loss functions and likelihood: Explain how minimizing a loss function can, for many modern models, be mathematically equivalent to maximizing the probability of observed data under an assumed statistical model.
Why it matters: It helps connect the language of modern model training to Fisher’s estimation principle.
Key terms: linear regression, maximum likelihood, normally distributed errors, loss functions, objective functions, likelihood, model parameters
3.5 Maximum likelihood and neural networks
This section connects neural-network training to maximum likelihood, showing how common losses express likelihood objectives even though neural networks require different optimization methods.
- Loss and likelihood equivalence: Explain why minimizing many common neural-network losses is equivalent to maximizing the likelihood of the observed data under an assumed probability model.
Why it matters: It reveals the statistical objective behind a familiar description of neural-network training. - Optimization at neural-network scale: Distinguish the shared likelihood objective from the different ways classical models and neural networks optimize it.
Why it matters: It separates what the model is trying to achieve from how it finds suitable parameters. - Task-specific likelihood losses: Describe the loss and probability-model pairings given for neural-network classification and regression.
Why it matters: These examples show how the likelihood principle appears in two common learning tasks.
Key terms: maximum likelihood, loss function, negative log-likelihood, closed-form solutions, gradient descent, network weights, cross-entropy loss, categorical probability model, mean squared error, normally distributed errors
3.6 MLE beyond regression analysis and neural networks
This section explains that maximum likelihood estimation extends beyond regression and neural networks because the same framework can be applied to many kinds of data once a probability model is specified.
- Breadth of likelihood estimation: Explain why maximum likelihood estimation is not limited to regression or neural networks: once a probability model is specified, its general estimation framework can apply to many forms of data.
Why it matters: It helps you recognize maximum likelihood as a general approach to estimation, not a technique tied to one model.
Key terms: maximum likelihood estimation, probability model, likelihood principle
3.6.1 Poisson and exponential models
This section distinguishes Poisson counts from exponential waiting times and shows how maximum likelihood estimates the event rate in each model.
- Counts versus waiting times: Distinguish the event-based data modeled by the Poisson and exponential distributions: counts within a fixed interval versus times between successive events.
Why it matters: The kind of observation determines which model—and which rate estimate—fits the data. - Maximum likelihood rate estimates: State how maximum likelihood estimates the event rate for Poisson counts and for exponential waiting times, and explain what the different formulas illustrate.
Why it matters: The same estimation principle produces different rate formulas for different kinds of event data.
Key terms: Poisson distribution, exponential distribution, fixed interval, time between successive events, maximum likelihood, event rate, sample mean, observed counts, observed waiting times, closed-form estimators
3.6.2 Naïve Bayes
The section shows how Naïve Bayes combines maximum-likelihood parameter estimation with Bayesian classification.
- Naïve Bayes estimation and inference: Explain how Naïve Bayes uses maximum likelihood to estimate probabilities from training data, then uses Bayes’ Theorem to calculate class posterior probabilities. Describe how conditional independence makes the joint likelihood a product of individual feature likelihoods.
Why it matters: It distinguishes how Naïve Bayes fits its probabilities from how it uses them to classify observations.
Key terms: Naive Bayes, maximum likelihood, conditionally independent, joint likelihood, posterior probability
3.6.3 Hidden Markov Models
This section explains how maximum likelihood estimation applies to sequential data with hidden states, using an iterative method rather than a closed-form solution.
- Hidden-state likelihood estimation: Explain why hidden states make parameter estimation in Hidden Markov Models challenging and how Baum-Welch iteratively increases the likelihood of observed sequences. Describe how this differs computationally from regression while retaining the same likelihood-based objective.
Why it matters: It shows how the likelihood principle still guides estimation when hidden states make a direct solution difficult.
Key terms: Hidden Markov Models, hidden variables, Baum-Welch algorithm, Expectation-Maximization (EM) algorithm, likelihood
3.7 Model Selection
Introduces model selection as choosing among models fitted to the same data while balancing fit against complexity.
- Model selection and complexity: Distinguish model selection from parameter estimation, and explain why choosing among models fitted to the same data requires balancing goodness of fit against complexity. State that maximum likelihood provides a foundation for this comparison, without describing the tools' mechanics.
Why it matters: It explains why the best-fitting model is not automatically the most appropriate one.
Key terms: model selection, parameter estimation, goodness of fit, model complexity, maximum likelihood
3.7.1 Likelihood Ratio Tests
This section explains how a likelihood ratio test compares nested models and judges whether added parameters improve the fit enough to keep them.
- Nested Model Comparison: Describe when a likelihood ratio test applies and what it asks about added parameters.
Why it matters: Recognizing nested models tells you when this comparison is appropriate. - Likelihood Improvement and Significance: Explain how the models’ maximized likelihoods inform the preference for the simpler or more complex model, and how statistical significance is assessed.
Why it matters: The likelihood improvement provides a basis for deciding whether added complexity is warranted.
Key terms: likelihood ratio test, nested models, added parameters, maximized likelihoods, maximum likelihood estimates, test statistic, approximate chi-square distribution, statistical significance
3.7.2 Akaike and Bayesian Information Criteria
This section explains how AIC and BIC use maximized likelihood and a complexity penalty to compare competing models, including models that are not nested.
- Likelihood and complexity tradeoff: Explain why information criteria offer an alternative when competing models are not nested, and how AIC and BIC balance a fitted model’s maximized likelihood against its complexity.
Why it matters: A better fit alone does not justify a more complicated model. - AIC and BIC priorities: Distinguish AIC’s emphasis on predictive performance from BIC’s stronger complexity penalty, especially as sample size increases.
Why it matters: The two criteria can favor different models because they weigh added complexity differently.
Key terms: Akaike Information Criterion, Bayesian Information Criterion, maximized likelihood, model complexity, parsimony, AIC, BIC, predictive performance, sample size
3.8 Fisher's enduring legacy
This section explains why Fisher’s maximum likelihood principle endures and how its approach to inference complements Bayesian reasoning.
- Enduring maximum likelihood principle: Explain that, given a probability model, maximum likelihood estimation chooses the parameter values that make the observed data most plausible, and that this same objective applies across different learning methods.
Why it matters: The same estimation principle connects models that otherwise look quite different. - Fisher and Bayesian perspectives: Distinguish Fisher’s direct estimation of parameters from observed data through likelihood from Bayesian updating of beliefs using new evidence and prior beliefs.
Why it matters: The distinction helps you recognize whether an inference method is estimating from data through likelihood or updating beliefs.
Key terms: maximum likelihood estimation, probability model, likelihood, observed data, Bayesian inference, prior beliefs, statistical inference
3.9 Summary
Where this is used later
- Why it matters (chapter 4) · Fisher’s approach extracts evidence from data.
- Cross-entropy as likelihood and information (chapter 11) · Connects neural-network cross-entropy to likelihood.
- Updating the network (chapter 11) · Develops how network parameters are updated during learning.
- Fisher: estimating unknown parameters through maximum likelihood (chapter 10) · The discussion of Fisher’s maximum-likelihood contribution draws on likelihood as a common estimation objective.
Concepts explained in this chapter
- maximum likelihood estimationA maximum likelihood estimate is a parameter value that maximizes a statistical model’s likelihood function.
- likelihoodA likelihood function measures how compatible observed data are with different parameter values of a statistical model.
- logistic regressionLogistic regression is a statistical model that estimates the probability of an outcome from a weighted combination of input variables.
- linear predictorA linear predictor is a weighted additive combination of input variables and coefficients that produces an intermediate model score.
- parameterA parameter is a quantity that determines aspects of a model’s behavior.
- sigmoidA sigmoid function is a smooth S-shaped function that maps real-valued inputs to values between zero and one.
- efficiencyEfficiency describes how small an estimator's sampling variance is relative to competing estimators or an information-based bound.
- sufficient statisticA sufficient statistic is a statistic that contains all information in a sample relevant to a specified parameter under a specified model.
- estimator consistencyConsistency is the property that an estimator converges to the true parameter value as the amount of data increases under specified conditions.
- squared errorSquared-error loss measures prediction error by squaring the difference between a predicted and observed value.
- estimatorAn estimator is a rule or statistic that maps observed data to an estimate of an unknown parameter.
- ordinary least squaresOrdinary least squares estimates regression parameters by minimizing the sum of squared differences between observed and predicted responses.
- linear regressionLinear regression is a statistical model that represents a continuous response as a linear function of predictors and model parameters.
- independent variableA feature is an input attribute whose values are considered by a predictive model when making predictions.
- Cramer-Rao lower boundThe Cramér–Rao lower bound is a theoretical lower limit on the variance of unbiased estimators under specified regularity conditions and a probability model.
- optimizationAn optimization method or algorithm is a procedure for searching for solutions that improve an objective function.
- neural networkA neural network is a parameterized computational model composed of interconnected layers that transform inputs through learned functions.
- hidden Markov modelA hidden Markov model is a probabilistic model of sequences in which hidden states evolve according to transition probabilities and generate observable outputs.
- AICThe Akaike information criterion is a model selection criterion that estimates relative information loss while balancing fit against model complexity.
- BICThe Bayesian information criterion is a likelihood-based model-selection score that penalizes model complexity, with a penalty that increases with sample size.
- likelihood ratio testA likelihood ratio test compares nested statistical models using their maximized likelihoods to assess whether the more complex model provides a significant improvement.
- naive bayes classifierA naive Bayes classifier is a probabilistic classification model that applies Bayes' theorem while assuming features are conditionally independent given the class.
- model selectionModel selection is the process of choosing among candidate models based on their assumptions, representations, and expected behavior.
- interceptA bias term is an additive constant that shifts a model's output independently of its input features.
- log-likelihoodA log-likelihood is the logarithm of a likelihood function, converting products of probabilities into sums for analysis and optimization.
- thresholdA decision threshold is a cutoff used to convert a score or estimated probability into a decision or predicted class.
- binary random variableA Bernoulli random variable has two possible values, conventionally 0 and 1.
- uncertaintyUncertainty is the lack of complete information about the state of a system, a quantity, or the outcome of an event.
- meanExpected value is the probability-weighted average of the possible numerical outcomes of a random variable.
- Bayesian inferenceBayesian inference updates prior probabilities or distributions about hypotheses or parameters with observed data.
- method of momentsThe method of moments estimates parameters by equating theoretical distribution moments with corresponding sample statistics.
- prior probabilityA prior distribution represents uncertainty about an unknown quantity before incorporating observed data.
- standard deviationThe square root of variance, measuring dispersion in the same units as the quantity described.
- varianceVariance is the model's sensitivity to changes in the training sample, reflected in how its behavior varies across samples.
- Fisher information matrixFisher information is a quantity measuring how much information a probability model and its data provide about an unknown parameter.
- cross entropy lossCross-entropy loss measures the discrepancy between target outcomes and predicted probabilities.
- negative log-likelihoodA negative log-likelihood is the negative logarithm of a likelihood, commonly minimized as an objective equivalent to maximizing likelihood.
- test statisticA test statistic is a numerical quantity calculated from data and used to assess a statistical hypothesis.
- model complexityModel complexity is the degree of structural or parametric elaboration in a model that affects its flexibility and risk of fitting noise.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
