liveBook · Manning

Consistency

As explained in AI Model Evaluation

Consistency is the degree to which an evaluator produces stable judgments across comparable inputs or repeated evaluations.

Consistency measures whether an evaluator gives stable judgments across repeated evaluations of unchanged inputs. It is distinct from correctness: an evaluator can be consistently wrong or correct on average but inconsistent from run to run. section 10.4 section 10.4.1

To test consistency, run the same prompt, model, and input multiple times. Five to ten runs per example is a reasonable starting point. Measure score spread with variance or standard deviation; for categorical verdicts, check how often the majority verdict holds across reruns. section 10.4.2

Repeated score

Five runs produce scores of 4, 4, 3, 4, and 5, with a mean of 4.0 and standard deviation of 0.63. Moderate variation may indicate that the rubric wording needs review. section 10.4.2

Instability can obscure small differences between model variants or make pass-rate gates unreliable. If identical inputs produce widely varying judgments, revisit the prompt or rubric before relying on aggregated scores or scaling evaluation. section 10.4.1 section 10.4.2

How AI Model Evaluation covers it

How deep it goes
Measures repeated-run score spread or categorical verdict agreement, recommending five to ten runs per example as a starting point.
Examples
Uses five scores of 4, 4, 3, 4, and 5, and describes risks to model comparisons and pass-rate gates.
What you can do after
Evaluate evaluator stability by rerunning identical inputs and measuring score spread or categorical verdict agreement.
How it connects
Distinguishes consistency from correctness and connects stability testing to human validation, repeated sampling, prompt design, and rubric wording.

Also explained in

Compare how 28 books explain consistency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.