A stability check tests whether an evaluator produces consistent judgments when the same input is evaluated repeatedly or under comparable conditions. For an LLM judge, agreement with human labels is necessary but not sufficient: a judge can be accurate on average and still fluctuate between runs, or be consistently wrong. section 10.4 section 10.4.1
The required stability depends on the decision. If two model variants score 4.1 and 4.3, but the judge can score the same response anywhere from 3 to 5, the apparent difference may be noise. Likewise, a pass rate that moves from 74% to 86% across runs is not a dependable basis for an 80% approval gate. section 10.4.1
Use repeated sampling to check a judge before relying on its scores:
- Select examples from the validation set. Run each through the same judge, with the identical prompt and input, several times; five to ten runs is a reasonable starting point. section 10.4.2
- For numerical scores, inspect variance or standard deviation per example. For categorical verdicts, measure how often the majority verdict holds across reruns. section 10.4.2
- Review unstable examples and the prompt or rubric for ambiguity before scaling. Repeated sampling tests consistency; using repeated samples as a panel instead aggregates them into a verdict. section 10.4.2
Repeated scores on a five-point scale
An example receives scores of 4, 4, 3, 4, and 5 over five runs. Its mean is 4.0 and its standard deviation is 0.63. The spread flags moderate variance and suggests reviewing the rubric wording; the mean alone hides that variation. section 10.4.2
Make stability checks part of recurring validation alongside canary sets. Repeated sampling surfaces general variance; swapped-order reruns surface position bias. High variance or position sensitivity can trigger escalation to a stronger judge or human reviewer. But uncertainty-based escalation can miss a judge that is confidently and consistently wrong, so it does not replace accuracy validation. section 10.7 section 10.5.2
