liveBook · Manning

Consistency

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Consistency is the degree to which judgments remain aligned across annotators, cohorts, and time under a shared rubric.

Consistency measures whether judgments or outputs stay aligned under a shared standard. In evaluation, it can mean factual alignment with a source, agreement across generated samples, or stability across annotators and cohorts. section 4.2.5 section 5.2.4 section 8.1.1

Use the form of consistency that matches the failure you need to diagnose:

  • Check factual consistency by testing whether every summary claim is entailed by the source; penalize hallucinated facts. section 4.2.5
  • Check cross-sample consistency by generating independent outputs: stable outputs suggest peaked token distributions, while varied outputs suggest uncertainty or absent knowledge. section 5.2.4
  • Check annotation consistency by comparing labeler agreement across cohorts and time, while tracking changes to the rubric. section 8.1.1
  • Choose sample counts by risk: use 3–5 for obvious variation, 10–20 for higher-stakes detection, and reconsider sampling above 20 when closed-domain verification is preferable. section 5.2.4

Financial research assistant

When support expands from single-quarter earnings questions to multi-quarter trends, new domain-trained labelers may interpret the rubric differently; separating policy, rubric, and pool effects helps locate quality regressions. section 8.1.1

Table 4.5 Impact of removing G-Eval components
Component Removed
Correlation Drop
Implications
GPT-4 → GPT-3.5
–0.136
Smarter models matter more than prompt tricks
Detailed evaluation criteria
-0.102
Write specific rubrics, not vague instructions
Chain-of-thought steps
-0.045
Worth the extra tokens for complex assessments
Probability weighting
-0.028
Nice-to-have if your API supports it

Compare how model capability, detailed criteria, reasoning steps, and probability weighting affect G-Eval correlation. section 4.2.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Distinguishes three operational forms and gives sample-count guidance, but does not define a universal consistency score.
Examples
Uses source-grounded summaries, repeated samples, and a financial assistant expanding from quarterly earnings to multi-quarter trends.
What it assumes
Assumes basic knowledge of summaries, language-model sampling, rubrics, and human annotation while distinguishing their consistency problems.
What you can do after
Diagnose hallucination risk or annotation-quality drift by comparing claims, samples, cohorts, rubrics, and production stakes.
How it connects
Connects factual consistency with hallucination detection, sampling with token probabilities, and annotation drift with reward models and RLHF.

Also explained in

Compare how 28 books explain consistency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.