Variance measures how much outcomes fluctuate around an average or expected value. In evaluation, high variance can make a real improvement hard to distinguish from noise, increasing the risk of both false positives and false negatives. section 6.5.4
AI experiment outcomes can fluctuate because of randomness, differences in user behavior, or noisy inputs—not necessarily because the model is better or worse. Subtle effects, long feedback loops, and rare conversions make meaningful differences harder to detect. Choose stable, relevant metrics, a sufficient sample size, and an evaluation window suited to the metric; highly variable or rare metrics may require more users or more time. section 6.5.4 section 6.3.2
A movie recommender
A personalization model may improve completion rates for heavy users while making little difference for casual users. An overall average can hide that concentrated gain, while a few binge users can dominate watch-time measurements and increase variance. Examine subgroups when the goal is to help them, but prioritize the overall effect when the product goal is to move the average. section 8.6.1
Figure 8.2 illustrates how raw averages can flatten heterogeneous effects. In this movie recommender example, heavy users show a clear lift in watch time under the new model, while casual users see little to no change. When averaged across all users, the treatment effect looks much smaller than it does for the subgroup that benefited most. This illustrates why top-line A/B results can mislead in AI evaluations without subgroup analysis or variance reduction.
Compare the heavy-user and casual-user results with the smaller top-line effect. section 8.6.1
An LLM judge adds another source of variance: the same response can receive different scores on repeated runs. If two variants average 4.1 and 4.3 but the same response can swing from 3 to 5, the judge’s instability may obscure the difference. A stable judge is not necessarily correct, so check stability separately from agreement with human labels. section 10.4.1 section 10.4.4
Make variance visible and actionable in judge-based evaluation:
- Repeat judgments on the same examples during calibration; compare judgments with human-labeled examples to check alignment separately. section 9.10.3
- Report a confidence interval or standard deviation alongside aggregate scores. Set an acceptable variance threshold before using a score to gate a launch or promotion. section 10.4.4
- If stability fails the threshold, revisit the prompt, add reasoning-first structure, or use panel-based aggregation. For pairwise judgments, track how often swapping response order changes the verdict. section 10.4.4
- When results change unexpectedly, rerun a canary set against its baseline, check prompt and rubric changes, and repeat stability and order-swap tests before concluding that the evaluated model regressed. section 10.6.2























