liveBook · Manning

Verification

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Claim verification is the process of checking an asserted claim against evidence or authoritative information to determine whether it is supported.

Claim verification checks whether each factual claim is supported by available evidence. It turns a generated response into independently testable assertions and exposes unsupported details instead of hiding them behind fluent prose. section 5.1.4 section 5.3.1

Use a claim-verification workflow:

  • Extract atomic claims, making each statement independently verifiable. section 5.3.1
  • Select evidence appropriate to the claim: retrieved documents, structured knowledge bases, or web search. section 5.3.1
  • Classify each claim as supported, contradicted, or lacking evidence. section 5.3.1
  • Aggregate the results into faithfulness, hallucination counts, or importance-weighted quality gates. section 5.4.1 section 6.3.1
  • Route failures to logging, human review, response blocking, or regeneration. section 5.4.1

Financial RAG response

Decomposing an answer into six claims about revenue, growth, cloud services, European expansion, a new CEO, and international revenue yields three supported and three unsupported claims. Faithfulness is therefore 3/6 = 0.50; the unsupported claims reveal fabricated details introduced when retrieved documents leave information gaps. section 6.3.1

Listing 5.4 Claim extraction using an LLM
CLAIM_EXTRACTION_PROMPT = """Extract all factual claims from the following text. Each claim should be a single, independently verifiable statement. Return one claim per line.
 
Text: {text}
 
Claims:"""
 
def extract_claims(text: str, model: str = “gpt-5.4”) -> list[str]:
    response = client.chat.completions.create(
        model=model,
        messages=[{
            "role": "user",
            "content": CLAIM_EXTRACTION_PROMPT.format(text=text)
        }],
        temperature=0
    )
    claims = response.choices[0].message.content.strip().split(“\n”)
    return [c.strip(“- “).strip() for c in claims if c.strip()]

Shows an LLM prompt and function for extracting independently verifiable claims. section 5.3.1

Listing 5.5 Claim verification against reference documents
VERIFICATION_PROMPT = """Given the following reference document and a factual claim, determine if the document SUPPORTS the claim, CONTRADICTS the claim, or provide NO EVIDENCE for the claim.
 
Reference Document:
{document}
 
Claims: {claim}
 
Verdict (SUPPORTS / CONTRADICTS / NO EVIDENCE):"""
 
def verify_claim(claim: str, documents: list[str], model: str = “gpt-5.4”) -> dict:
    verdicts = []
    for doc in documents:
        response = client.chat.completions.create(
            model=model,
            messages=[{
                "role": "user",
                "content": VERIFICATION_PROMPT.format(document=doc, claim=claim)
            }],
            temperature=0,
            max_tokens=10
        )
        verdict = response.choices[0].message.content.strip()
        verdicts.append(verdict)
 
    supported = sum(1 for v in verdicts if “SUPPORTS” in v)
    contradicted = sum(1 for v in verdicts if “CONTRADICTS” in v)
 
    return {
        “claim”: claim,
        “supported”: supported,
        “contradicted”: contradicted,
        “no_evidence”: len(verdicts) - supported - contradicted,
        “is_hallucination”: contradicted > 0 or supported == 0          #A
    }

Shows reference-document verification with SUPPORTS, CONTRADICTS, and NO EVIDENCE verdicts. section 5.3.1

Listing 5.6 Computing faithfulness from verified claims
def faithfulness(text: str, reference_docs: list[str]) -> dict:
    claims = extract_claims(text)
 
    results = []
    for claim in claims:
        verification = verify_claim(claim, reference_docs)
        results.append(verification)
 
    supported = sum(1 for r in results if not r[‘is_hallucination’])
    total = len(results)
 
    faithfulness = supported / total if total > 0 else 1.0
 
    return {
        “faithfulness”: faithfulness,
        “total_claims”: total,
        “supported_claims”: supported,
        “hallucinated_claims”: total - supported,
        “details”: results                                                     #A
    }

Shows how verified claims become faithfulness, support, hallucination, and per-claim detail metrics. section 5.3.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Decomposes responses, selects evidence, assigns verdicts, computes metrics, and connects failures to operational actions.
Examples
Uses arXiv papers, Wikidata facts, web-search queries, and a financial RAG answer with six decomposed claims.
What it assumes
Assumes generated text and evidence sources, while teaching claim decomposition and verification from the ground up.
What you can do after
Apply it via LLM prompts and retrieved documents to produce claim-level verdicts and faithfulness reports.
How it connects
Connects claim extraction with faithfulness, hallucination detection, retrieval-augmented generation, natural language inference, and knowledge bases.

Also explained in

Compare how 9 books explain verification →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.