Fisher information measures the curvature of a model’s log likelihood with respect to its parameters. For a model with K parameters, I(θ) is a K × K matrix. Its inverse provides a variance limit for unbiased parameter estimates, making it useful when choosing which data to label in active learning. section 7.3.1 section 7.8
The motivation starts with mean squared error: it combines squared bias and variance, so reducing variance matters even though a biased estimate can sometimes have lower error. Under the stated regularity condition, the Cramér–Rao lower bound relates the variance of an unbiased estimator to inverse Fisher information. An active learner seeking lower parameter variance therefore favors queries that increase information. section 7.3.1 section 7.8
For multiple parameters, query selection needs a criterion for the inverse information matrix:
- A-optimality minimizes its trace, Tr(I⁻¹(θ)). section 7.3.1
- D-optimality minimizes its determinant, |I⁻¹(θ)|. section 7.3.1
- E-optimality minimizes its largest eigenvalue, λmax(I⁻¹(θ)). section 7.3.1
This variance-reduction approach has a computational cost: inverting a K × K matrix for each of U unlabeled candidates takes O(UK³) time. It can therefore be slower than simpler query strategies such as uncertainty sampling. section 7.3.1
