Sampling selects outcomes from a probability distribution rather than always choosing the most likely outcome. In evaluation, it supports both human auditing and hallucination detection. section 4.4.4 section 5.2.4
Use sampling in two distinct ways:
- Route a random percentage of automated evaluations to human annotators to estimate judge accuracy across the full evaluation distribution. section 4.4.4
- Generate independent responses with temperature greater than zero, then compare them with a target response for consistency. section 5.2.2
- Treat repeated consistency as evidence rather than proof: peaked token distributions tend to produce stable outputs, while flatter distributions produce variation. section 5.2.4
Checking a factual response
Given a response about the Pantheon, generate 5–10 independent reference responses for development or 10–20 for offline evaluation. Compare each target sentence with the samples: repeated claims appear more factual, while contradictory or absent claims appear more likely to be hallucinated. section 5.2.2 section 5.2.4
Figure 5.5 The SelfCheckGPT pipeline. The same prompt produces one target response and N stochastic examples. Each sentence in the target is checked for consistency against the samples. Consistent sentences score as factual; inconsistent sentences score as hallucinated.
Follow the pipeline from one target response to independent stochastic samples and sentence-level consistency scores. section 5.2.2
Choose the sample count according to error tolerance and cost. Three to five samples can expose obvious fabrications; ten to twenty better support high-stakes detection, while larger sets bring diminishing returns. section 5.2.4
Account for limitations: real-time production overhead can be prohibitive, sentence-level checks can hide sub-sentence errors, and systems without temperature or related controls cannot use this technique. section 5.2.5






























