Maximum Likelihood Estimation (MLE)

Maximum Likelihood Estimation (MLE) is a fundamental technique used in statistics to estimate the parameters of a statistical model. It is particularly useful in the context of Gaussian distributions, where it helps in determining the mean and variance from training data. The goal of MLE is to find the parameter values that maximize the likelihood of the observed data.

Overview

MLE is a method that estimates the parameters of a statistical model by maximizing the likelihood function. In the context of Gaussian distributions, MLE is used to estimate the mean (μ) and variance (σ) that maximize the likelihood of the observed data. This involves maximizing the log-likelihood function, which, for Gaussian distributions, has a closed-form solution. The MLE solutions for Gaussian distributions coincide with the sample mean and variance, allowing for the prediction of probabilities for specified ranges.

Example

Consider an example where MLE is used to estimate the mean and standard deviation of a small sample of height data. The MLE results in a mean of 149.68 and a standard deviation of 11.52. These estimates differ from the true values due to the small sample size, highlighting a limitation of MLE when applied to limited data.

Maximum Likelihood Estimate for a Gaussian

The following listing demonstrates how to compute the maximum likelihood estimate for a Gaussian distribution:

Listing 6.4 Maximum likelihood estimate for a Gaussian

sample_mean = X.mean()                          #1

sample_std = X.std()

gaussian_mle = Normal(sample_mean, sample_std)  #2

a, b = torch.Tensor([160]), torch.Tensor([170]) #3

prob = gaussian_mle.cdf(b) - gaussian_mle.cdf(a)
#1 Estimates Gaussian MLE parameters <span class="times"><img alt="" class="calibre9" height="12px" src="../Images/AR_micro.png" /></span> and <span class="timesbold">Σ</span>. They equal the sample mean and sample covariance of the training data. See equations <a style="text-decoration: none; color: #4080C0;" href="../Text/06.html#eq-gauss-MLE-mean">6.25</a> and <a style="text-decoration: none; color: #4080C0;" href="../Text/06.html#eq-gauss-MLE-covar">6.26</a>.
#2 Defines a Gaussian with the estimated parameters
#3 Once the Gaussian is estimated, we can use it to predict probabilities.

In this listing, the sample mean and standard deviation are calculated from the data X. These values are then used to define a Gaussian distribution (gaussian_mle). The cumulative distribution function (CDF) of this Gaussian is used to predict the probability of a specified range, in this case, between 160 and 170.

Conclusion

Maximum Likelihood Estimation is a powerful tool for parameter estimation in statistical models, especially for Gaussian distributions. However, its reliability can be affected by the size of the sample, as demonstrated in the example with height data. Despite this limitation, MLE remains a widely used method due to its theoretical properties and practical applicability.