A fluent answer can contain invented facts even when it sounds plausible. SelfCheckGPT estimates hallucination risk by comparing an answer with independently sampled responses to the same prompt. It needs no external knowledge base or access to model internals, making it useful for open-domain, black-box models. Agreement is a signal of reliability, not proof of truth. section 5.0 section 5.2 section 5.2.5
To check a response:
- Generate the target response to the prompt. section 5.2.2
- Generate N additional responses independently from the same prompt, using a temperature above zero. Do not expose any sample to the target or other samples. section 5.2.2
- Compare each target sentence with the samples. Consistent sentences receive lower hallucination-risk scores; contradictions or missing support can raise the score, depending on the comparison method. section 5.2.2 section 5.2.3
Start with 5–10 samples for a small development test or 10–20 for offline evaluation. More samples can expose subtler variation, but each adds generation cost and latency. The original evaluation used 20 samples. section 5.2.2 section 5.2.4
One implementation uses BERTScore to compare each target sentence with every sentence in a sample. It takes the best match from each sample, averages those matches, and subtracts the average from 1. Higher scores indicate greater estimated risk. BERTScore measures semantic similarity, however, and can miss contradictions between similar statements. section 5.2.3
Scoring a sentence
If the best matches for a target sentence score 0.90, 0.80, and 0.40 across three samples, their average is 0.70. The inverted hallucination-risk score is 1 − 0.70 = 0.30. This is a screening signal, not a verified probability that the sentence is false. section 5.2.3
Table 5.2 Comparison of SelfCheckGPT variants
|
Variant
|
Accuracy
|
Cost
|
Latency
|
Best for
|
|
BERTScore
|
Moderate
|
Low
|
Fast
|
High-volume screening
|
|
NLI
|
Good
|
Low-Medium
|
Medium
|
Detecting contradictions
|
|
QA
|
Good
|
Medium
|
Medium
|
Fine-grained claim verifications
|
|
LLM-as-a-Judge
|
Highest
|
High
|
Slow
|
High-stakes evaluation
|
Compare BERTScore, natural language inference, question answering, and LLM-as-a-judge variants by their reported cost, latency, and use cases. section 5.2.3
The method can miss a false belief repeated consistently across samples. It can also flag a whole sentence without identifying which embedded claim is unsupported. For auditable verification, decompose responses into claims and check them against evidence when trusted evidence is available. section 5.2.5 section 5.1.4
In an evaluation on WikiBio, the LLM-as-a-judge variant achieved 0.832 sentence-level AUC-PR, a measure of precision and recall across detection thresholds. This result is specific to that evaluation, not a guarantee for every deployment. section 5.2.5
