liveBook · Manning

Fidelity

As explained in Retrieval Augmented Generation, The Foundational Ideas

Faithfulness measures whether a generated response is supported by the retrieved context rather than introducing unsupported claims.

A RAG answer can address the question and still invent a policy. In a mortgage compliance assistant, improved retrieval was followed by a drop in faithfulness: the right documents came back, but answers drifted away from them. Fidelity, measured here as faithfulness, asks whether each factual claim in the generated answer is supported by the retrieved context. section 14.0 section 14.2.1

RAGAS measures faithfulness without reference answers. An LLM first splits the answer into factual claims, then checks each claim against the retrieved passages with a supported or unsupported verdict. The score is supported claims divided by total claims. If three of four claims have support, the score is 0.75; the unsupported claim still needs investigation. section 14.2 section 14.2.1

An on-topic answer with a fabricated detail

An advisor asks whether a client denied six months ago can reapply. The answer makes four claims; the retrieved documents support three, but say nothing about the answer’s promise of automatic approval. Faithfulness is 3/4, or 0.75. Answer relevance can remain high because the response addresses the question—it does not establish that the promise is grounded. section 14.2.4

Read the score alongside the other RAGAS dimensions:

  • Context relevance asks whether retrieval returned useful material; low context relevance points toward a retrieval problem. section 14.2
  • Faithfulness asks whether generation stays with that material; low faithfulness with adequate context relevance points toward hallucination. section 15.5.2
  • Answer relevance asks whether the response addresses the question. It cannot substitute for a support check: in a two-answer comparison, the fabricated answer scored higher on answer relevance but zero on faithfulness. section 14.2.4
Listing 14.5 RAGAS-style faithfulness judge.
import json
 
class FaithfulnessJudge:
    """Evaluate if an answer is grounded in retrieved context."""
 
    def __init__(self, llm_client):
        """Initialize with an LLM client interface."""
        self.llm = llm_client
 
    def _extract_claims(self, answer: str) -> list[str]:
        """Break an answer into individual factual claims."""
        prompt = (  #A
            "Break this answer into independent factual "
            f"claims, one per line:\n{answer}"
        )
        response = self.llm.generate(prompt)
        return [c.strip() for c in response.split("\n") if c.strip()]
 
    def evaluate(self, context: str, answer: str) -> dict:
        """Return faithfulness score and per-claim verdicts."""
        claims = self._extract_claims(answer)  #B
        if not claims:  #C
            return {"score": 0.0, "claims": [], "error": "no claims"}
        verdicts = []
        for claim in claims:  #D
            prompt = (
                f"Context: {context}\nClaim: {claim}\n"
                "Is this claim supported by the context? "
                "Reply 'Yes' or 'No' with a brief reason."
            )
            result = self.llm.generate(prompt)
            supported = result.strip().lower().startswith("yes")
            verdicts.append({
                "claim": claim,
                "supported": supported,
                "reasoning": result,
            })
        score = sum(v["supported"] for v in verdicts) / len(verdicts)
        return {"score": score, "claims": verdicts}  #E
 

The RAGAS-style judge shows the two calls behind the metric: extract independent claims, then verify them against context. Its per-claim verdicts identify what was fabricated, rather than leaving only an aggregate score. section 14.5.1

Do not treat one score or a passing threshold as proof of reliability. LLM judges can favor familiar phrasing and their own output; compare results over time and investigate the claims behind a change. In production, retain retrieved context with responses so failures can be traced to retrieval or generation, and sample queries regularly to watch for faithfulness trends. section 14.0 section 15.6.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Works through atomic claim extraction, context verification, score calculation, per-claim debugging, and an LLM-judge implementation.
Examples
Uses a reapplication answer with one unsupported automatic-approval claim and compares grounded and fabricated compliance answers.
What it assumes
Assumes familiarity with generated answers and retrieved context; explains claim-level scoring without requiring reference answers.
What you can do after
Diagnose/debug with it by inspecting unsupported claims and distinguishing generation drift from retrieval problems.
How it connects
Connects to RAGAS, hallucination, context relevance, answer relevance, LLM-as-judge evaluation, and production monitoring.

Also explained in

Compare how 11 books explain fidelity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.