liveBook · Manning

Likelihood

As explained in Timeless Algorithms, The Foundational Ideas

A likelihood function measures how compatible observed data are with different parameter values of a statistical model.

Likelihood compares possible parameter values by how well an assumed probability model explains observed data. For fixed observations D, write L(θ) = p(D | θ): the data remain fixed while the parameter θ varies. Probability instead considers possible data under a fixed parameter. For continuous observations, the likelihood is based on a density rather than the probability of an exact value. section 3.1 section 3.1.4

Maximum likelihood estimation (MLE) selects a parameter value that maximizes L(θ). Fisher’s approach treats the parameter as fixed but unknown, rather than assigning it a prior distribution. The estimate identifies the value most compatible with the sample under the model; it does not establish the true value. section 3.0 section 10.2.2

A coin and a normal mean

If a coin lands heads seven times in 10 independent tosses with a constant probability of heads, the maximum likelihood estimate is p = 0.7. The small sample and the assumptions limit the conclusion. For independent normal observations with unknown mean and known variance, maximizing the log-likelihood instead yields the sample mean: taking logs turns the product of densities into a sum, and the maximizing value is unchanged. section 10.2.2 section 3.1.4

Likelihood also enters Bayesian inference, but it is not itself a posterior probability. Bayes’ theorem combines the likelihood P(D | H) with the prior P(H), then normalizes by the evidence P(D), to obtain P(H | D). The prior can matter greatly: a positive test with 99% sensitivity and a 1% false-positive rate gives only about a 9% disease probability when prevalence is 1 in 1,000. section 2.2.1 section 2.2.2

For independent binary outcomes yᵢ with predicted probabilities pᵢ, the likelihood is L = ∏ᵢ pᵢ^yᵢ(1 − pᵢ)^(1 − yᵢ). Each observation contributes pᵢ when yᵢ = 1 and 1 − pᵢ when yᵢ = 0. Maximizing its log is equivalent to minimizing binary cross-entropy, linking likelihood to logistic regression and classification-network training. section 3.3.2 section 11.4.2

Results depend on the model and its assumptions. For example, with independent normally distributed errors and fixed variance, maximizing the regression likelihood gives the same coefficients as minimizing squared residuals. A likelihood-based estimate should still be evaluated for how reliably it recovers the unknown parameter across repeated samples. section 3.4.2 section 3.1.5

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Develops likelihood from probability models through log-likelihood optimization, including discrete and continuous worked examples.
Examples
Works through coin tosses, a rare-disease test, a normal mean, and binary classification.
What you can do after
Evaluate and understand parameter estimates and classification objectives in light of their probability models and assumptions.
How it connects
Connects likelihood to maximum likelihood estimation, Bayesian inference, logistic regression, and cross-entropy.

Also explained in

Compare how 7 books explain likelihood →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.