liveBook · Manning

Metric stability

As explained in Evaluating AI Systems

The degree to which a metric's value remains similar when the evaluation dataset changes slightly.

Metric stability measures whether small, reasonable changes to an evaluation dataset cause large score swings. Large swings suggest noisy results that cannot support confident conclusions. section 4.3.2

Probe stability in several ways:

  • Check confidence intervals; wide intervals indicate uncertainty. section 4.3.2
  • Recalculate the metric across random subsets; tightly clustered scores indicate robustness. section 4.3.2
  • Use systematic sensitivity analysis to test how dataset changes affect the metric. section 4.3.2
Figure 4.7 Using metric convergence to assess evaluation dataset size. As more examples are added, metric estimates become increasingly stable, helping identify the point at which additional data yields diminishing returns.

Metric convergence as examples accumulate helps identify when estimates stabilize and additional data yields diminishing returns. section 4.3.2

How Evaluating AI Systems covers it

How deep it goes
Defines stability and outlines confidence intervals, resampling, and sensitivity analysis as checks.
Examples
Illustrates uncertainty with an accuracy estimate of 82% ± 9% and uses metric convergence as a visual.
What you can do after
Evaluate whether metric estimates remain robust under reasonable changes to the evaluation data.
How it connects
Connects evaluation dataset size with confidence intervals, resampling, and sensitivity analysis.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.