Consistency measures whether judgments or outputs stay aligned under a shared standard. In evaluation, it can mean factual alignment with a source, agreement across generated samples, or stability across annotators and cohorts. section 4.2.5 section 5.2.4 section 8.1.1
Use the form of consistency that matches the failure you need to diagnose:
- Check factual consistency by testing whether every summary claim is entailed by the source; penalize hallucinated facts. section 4.2.5
- Check cross-sample consistency by generating independent outputs: stable outputs suggest peaked token distributions, while varied outputs suggest uncertainty or absent knowledge. section 5.2.4
- Check annotation consistency by comparing labeler agreement across cohorts and time, while tracking changes to the rubric. section 8.1.1
- Choose sample counts by risk: use 3–5 for obvious variation, 10–20 for higher-stakes detection, and reconsider sampling above 20 when closed-domain verification is preferable. section 5.2.4
Financial research assistant
When support expands from single-quarter earnings questions to multi-quarter trends, new domain-trained labelers may interpret the rubric differently; separating policy, rubric, and pool effects helps locate quality regressions. section 8.1.1
Table 4.5 Impact of removing G-Eval components
|
Component Removed
|
Correlation Drop
|
Implications
|
|
GPT-4
→
GPT-3.5
|
–0.136
|
Smarter models matter more than prompt tricks
|
|
Detailed evaluation criteria
|
-0.102
|
Write specific rubrics, not vague instructions
|
|
Chain-of-thought steps
|
-0.045
|
Worth the extra tokens for complex assessments
|
|
Probability weighting
|
-0.028
|
Nice-to-have if your API supports it
|
Compare how model capability, detailed criteria, reasoning steps, and probability weighting affect G-Eval correlation. section 4.2.5



























