liveBook · Manning

Variance

As explained in AI Model Evaluation

Variance is a measure of how much a set of observed outcomes fluctuates around its expected or average value.

Variance measures how much outcomes fluctuate around an average or expected value. In evaluation, high variance can make a real improvement hard to distinguish from noise, increasing the risk of both false positives and false negatives. section 6.5.4

AI experiment outcomes can fluctuate because of randomness, differences in user behavior, or noisy inputs—not necessarily because the model is better or worse. Subtle effects, long feedback loops, and rare conversions make meaningful differences harder to detect. Choose stable, relevant metrics, a sufficient sample size, and an evaluation window suited to the metric; highly variable or rare metrics may require more users or more time. section 6.5.4 section 6.3.2

A movie recommender

A personalization model may improve completion rates for heavy users while making little difference for casual users. An overall average can hide that concentrated gain, while a few binge users can dominate watch-time measurements and increase variance. Examine subgroups when the goal is to help them, but prioritize the overall effect when the product goal is to move the average. section 8.6.1

Figure 8.2 illustrates how raw averages can flatten heterogeneous effects. In this movie recommender example, heavy users show a clear lift in watch time under the new model, while casual users see little to no change. When averaged across all users, the treatment effect looks much smaller than it does for the subgroup that benefited most. This illustrates why top-line A/B results can mislead in AI evaluations without subgroup analysis or variance reduction.

Compare the heavy-user and casual-user results with the smaller top-line effect. section 8.6.1

An LLM judge adds another source of variance: the same response can receive different scores on repeated runs. If two variants average 4.1 and 4.3 but the same response can swing from 3 to 5, the judge’s instability may obscure the difference. A stable judge is not necessarily correct, so check stability separately from agreement with human labels. section 10.4.1 section 10.4.4

Make variance visible and actionable in judge-based evaluation:

  • Repeat judgments on the same examples during calibration; compare judgments with human-labeled examples to check alignment separately. section 9.10.3
  • Report a confidence interval or standard deviation alongside aggregate scores. Set an acceptable variance threshold before using a score to gate a launch or promotion. section 10.4.4
  • If stability fails the threshold, revisit the prompt, add reasoning-first structure, or use panel-based aggregation. For pairwise judgments, track how often swapping response order changes the verdict. section 10.4.4
  • When results change unexpectedly, rerun a canary set against its baseline, check prompt and rubric changes, and repeat stability and order-swap tests before concluding that the evaluated model regressed. section 10.6.2

How AI Model Evaluation covers it

How deep it goes
Defines high variance and provides experiment-design, monitoring, and triage guidance without deriving a variance formula.
Examples
Illustrates variability with movie-recommender watch time and repeated LLM-judge scores for model variants.
What you can do after
Diagnose evaluation noise by repeating judge tests, checking order sensitivity, and comparing canary results with a baseline.
How it connects
Connects variance to A/B testing, sample size, subgroup analysis, LLM-as-a-judge, and stability testing.

Also explained in

Compare how 24 books explain variance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.