Maximum likelihood estimation (MLE) chooses model parameters θ that maximize the likelihood of the observed training data, p(X|θ). After selecting a probability-distribution family, MLE fits its unknown parameters using only the data. Geometrically, it seeks a distribution whose sampled point cloud overlaps the training-data cloud as much as possible. section 6.6.2 section 13.1.1
Likelihoods over independent examples are products, so optimization commonly maximizes log-likelihood instead. Because logarithm preserves the maximizer while simplifying products and exponential expressions, MLE can also be posed as minimizing negative log-likelihood. section 6.6.1 section 6.8.2
The fitting procedure depends on the model family:
- For a Gaussian, MLE has a closed-form solution: the fitted mean and variance are the training set's sample mean and sample variance; the multidimensional counterpart uses sample covariance. section 6.8 section 13.2
- For Gaussian mixture models, MLE has no closed-form parameter solution and instead requires iterative estimation. section 6.6.2 section 6.10
- For supervised networks, maximizing training-data likelihood corresponds to choosing parameters that minimize a loss when output mismatch lowers the modeled probability. section 9.3.5
Fitting a height distribution
Given observed heights of adult female residents and a Gaussian assumption, estimate μ and σ from the samples. The resulting normal distribution can then estimate an expected height or the probability that a height falls between 160 and 170 cm. With few samples, these MLE estimates can differ substantially from the underlying mean and standard deviation. section 6.8 section 6.8.1 section 13.8.1
A gradient-descent formulation minimizes Gaussian negative log-likelihood, but the illustrated optimization can diverge when μ and σ start far from expected values. MAP adds a prior-based regularization term, unlike MLE's data-only objective. section 6.8.2 section 6.10
Listing 6.4 Maximum likelihood estimate for a Gaussian
sample_mean = X.mean() #1 sample_std = X.std() gaussian_mle = Normal(sample_mean, sample_std) #2 a, b = torch.Tensor([160]), torch.Tensor([170]) #3 prob = gaussian_mle.cdf(b) - gaussian_mle.cdf(a)
A compact PyTorch implementation computes the sample mean and standard deviation, constructs a Normal distribution, and obtains an interval probability from CDF values. section 6.8.1
