An estimator is a rule or statistic that maps observed data to an estimate of an unknown parameter. Its quality depends on how it behaves as an estimate, not merely on producing a value. section 3.2
Three criteria characterize a good estimator:
- Consistency: the estimate converges toward the true parameter as the amount of data increases. section 3.2
- Efficiency: the estimator makes effective use of available information; for unbiased estimators, efficiency means attaining the smallest possible variance. section 3.2 section 3.9
- Sufficiency: the estimator retains all sample information relevant to the parameter, allowing data reduction without sacrificing inferential accuracy. section 3.2 section 3.9
Maximum likelihood estimation (MLE) specifies a probability model and chooses parameter values that maximize the likelihood of the observed data. Under appropriate regularity conditions, it can yield estimators with desirable properties. Method of moments offers another approach, while Bayesian inference provides a distinct framework. section 3.2.4 section 3.9
Regression
Logistic regression estimates coefficients by maximizing the likelihood of observed binary outcomes. Linear regression with normally distributed errors illustrates a case where maximizing likelihood is equivalent to minimizing the sum of squared errors. section 3.9




