Self-consistency measures agreement among independently sampled model outputs for the same input. Stable knowledge tends to produce repeatable answers, while hallucinated details often vary. Agreement is evidence of reliability, not proof of factuality. section 5.2.1
Apply the SelfCheckGPT-style procedure in three steps:
- Generate the response whose factuality you want to assess. section 5.2.2
- Sample 5–10 additional responses for development tests or 10–20 for offline evaluation, using temperature greater than zero and independent generations. section 5.2.2
- Compare each target sentence with the samples; repeated claims appear more factual, while absent or contradictory claims appear more hallucination-prone. section 5.2.2
Contrasting factual and unstable answers
Repeatedly asking for the author of the 2020 paper “GLU Variants Improve Transformer” yields “Noam Shazeer” consistently. Asking about co-authors of a 2019 sparse-mixture-of-experts paper may produce varying researcher names, signaling unreliable knowledge. The same sentence-level check can be applied to a response about the Pantheon in Rome. section 5.2.1 section 5.2.2
Figure 5.5 The SelfCheckGPT pipeline. The same prompt produces one target response and N stochastic examples. Each sentence in the target is checked for consistency against the samples. Consistent sentences score as factual; inconsistent sentences score as hallucinated.
Shows one target response, N stochastic samples, and sentence-level consistency checks that classify claims as more factual or hallucination-prone. section 5.2.2







