Consistency measures whether an evaluator gives stable judgments across repeated evaluations of unchanged inputs. It is distinct from correctness: an evaluator can be consistently wrong or correct on average but inconsistent from run to run. section 10.4 section 10.4.1
To test consistency, run the same prompt, model, and input multiple times. Five to ten runs per example is a reasonable starting point. Measure score spread with variance or standard deviation; for categorical verdicts, check how often the majority verdict holds across reruns. section 10.4.2
Repeated score
Five runs produce scores of 4, 4, 3, 4, and 5, with a mean of 4.0 and standard deviation of 0.63. Moderate variation may indicate that the rubric wording needs review. section 10.4.2
Instability can obscure small differences between model variants or make pass-rate gates unreliable. If identical inputs produce widely varying judgments, revisit the prompt or rubric before relying on aggregated scores or scaling evaluation. section 10.4.1 section 10.4.2



























