Metric stability measures whether small, reasonable changes to an evaluation dataset cause large score swings. Large swings suggest noisy results that cannot support confident conclusions. section 4.3.2
Probe stability in several ways:
- Check confidence intervals; wide intervals indicate uncertainty. section 4.3.2
- Recalculate the metric across random subsets; tightly clustered scores indicate robustness. section 4.3.2
- Use systematic sensitivity analysis to test how dataset changes affect the metric. section 4.3.2
Figure 4.7 Using metric convergence to assess evaluation dataset size. As more examples are added, metric estimates become increasingly stable, helping identify the point at which additional data yields diminishing returns.

Metric convergence as examples accumulate helps identify when estimates stabilize and additional data yields diminishing returns. section 4.3.2
