A RAG answer can address the question and still invent a policy. In a mortgage compliance assistant, improved retrieval was followed by a drop in faithfulness: the right documents came back, but answers drifted away from them. Fidelity, measured here as faithfulness, asks whether each factual claim in the generated answer is supported by the retrieved context. section 14.0 section 14.2.1
RAGAS measures faithfulness without reference answers. An LLM first splits the answer into factual claims, then checks each claim against the retrieved passages with a supported or unsupported verdict. The score is supported claims divided by total claims. If three of four claims have support, the score is 0.75; the unsupported claim still needs investigation. section 14.2 section 14.2.1
An on-topic answer with a fabricated detail
An advisor asks whether a client denied six months ago can reapply. The answer makes four claims; the retrieved documents support three, but say nothing about the answer’s promise of automatic approval. Faithfulness is 3/4, or 0.75. Answer relevance can remain high because the response addresses the question—it does not establish that the promise is grounded. section 14.2.4
Read the score alongside the other RAGAS dimensions:
- Context relevance asks whether retrieval returned useful material; low context relevance points toward a retrieval problem. section 14.2
- Faithfulness asks whether generation stays with that material; low faithfulness with adequate context relevance points toward hallucination. section 15.5.2
- Answer relevance asks whether the response addresses the question. It cannot substitute for a support check: in a two-answer comparison, the fabricated answer scored higher on answer relevance but zero on faithfulness. section 14.2.4
Listing 14.5 RAGAS-style faithfulness judge.
import json
class FaithfulnessJudge:
"""Evaluate if an answer is grounded in retrieved context."""
def __init__(self, llm_client):
"""Initialize with an LLM client interface."""
self.llm = llm_client
def _extract_claims(self, answer: str) -> list[str]:
"""Break an answer into individual factual claims."""
prompt = ( #A
"Break this answer into independent factual "
f"claims, one per line:\n{answer}"
)
response = self.llm.generate(prompt)
return [c.strip() for c in response.split("\n") if c.strip()]
def evaluate(self, context: str, answer: str) -> dict:
"""Return faithfulness score and per-claim verdicts."""
claims = self._extract_claims(answer) #B
if not claims: #C
return {"score": 0.0, "claims": [], "error": "no claims"}
verdicts = []
for claim in claims: #D
prompt = (
f"Context: {context}\nClaim: {claim}\n"
"Is this claim supported by the context? "
"Reply 'Yes' or 'No' with a brief reason."
)
result = self.llm.generate(prompt)
supported = result.strip().lower().startswith("yes")
verdicts.append({
"claim": claim,
"supported": supported,
"reasoning": result,
})
score = sum(v["supported"] for v in verdicts) / len(verdicts)
return {"score": score, "claims": verdicts} #E
The RAGAS-style judge shows the two calls behind the metric: extract independent claims, then verify them against context. Its per-claim verdicts identify what was fabricated, rather than leaving only an aggregate score. section 14.5.1
Do not treat one score or a passing threshold as proof of reliability. LLM judges can favor familiar phrasing and their own output; compare results over time and investigate the claims behind a change. In production, retain retrieved context with responses so failures can be traced to retrieval or generation, and sample queries regularly to watch for faithfulness trends. section 14.0 section 15.6.1










