liveBook · Manning

Self-consistency

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Self-consistency is agreement among independently sampled outputs from a model for the same input.

Self-consistency measures agreement among independently sampled model outputs for the same input. Stable knowledge tends to produce repeatable answers, while hallucinated details often vary. Agreement is evidence of reliability, not proof of factuality. section 5.2.1

Apply the SelfCheckGPT-style procedure in three steps:

  • Generate the response whose factuality you want to assess. section 5.2.2
  • Sample 5–10 additional responses for development tests or 10–20 for offline evaluation, using temperature greater than zero and independent generations. section 5.2.2
  • Compare each target sentence with the samples; repeated claims appear more factual, while absent or contradictory claims appear more hallucination-prone. section 5.2.2

Contrasting factual and unstable answers

Repeatedly asking for the author of the 2020 paper “GLU Variants Improve Transformer” yields “Noam Shazeer” consistently. Asking about co-authors of a 2019 sparse-mixture-of-experts paper may produce varying researcher names, signaling unreliable knowledge. The same sentence-level check can be applied to a response about the Pantheon in Rome. section 5.2.1 section 5.2.2

Figure 5.5 The SelfCheckGPT pipeline. The same prompt produces one target response and N stochastic examples. Each sentence in the target is checked for consistency against the samples. Consistent sentences score as factual; inconsistent sentences score as hallucinated.

Shows one target response, N stochastic samples, and sentence-level consistency checks that classify claims as more factual or hallucination-prone. section 5.2.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains intuition, sampling conditions, sentence comparison, and evidential limits without proving factuality.
Examples
Uses paper-authorship questions and a Pantheon description to contrast stable facts with variable generated claims.
What it assumes
Assumes a model that can generate multiple responses and explains temperature and independent sampling from the basics.
What you can do after
Diagnose hallucination risk by sampling independent responses and comparing target claims for consistency.
How it connects
Connects hallucination detection with factuality assessment and token-level confidence.

Also explained in

Compare how 8 books explain self-consistency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.