liveBook · Manning

Fidelity

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Faithfulness is a metric representing how accurately generated content reflects factual evidence or supported source information.

A plausible answer can mix supported facts with invented details. Faithfulness measures the proportion of an answer’s claims supported by evidence. A score near 1 means most claims are supported; a score near 0 means few are. It is a measure of grounding, not a complete measure of answer quality. section 1.1.3 section 6.3.1

To calculate faithfulness against a source:

A financial research answer

An answer makes six claims about a company. The retrieved earnings report supports its revenue, growth rate, and cloud-services explanation, but does not mention European expansion, a new CEO, or international revenue growth. Faithfulness is 3/6, or 0.50. The unsupported details may sound plausible, but the retrieved evidence does not establish them. section 6.3.1

Choose the verification target carefully. In a closed-domain task, such as document question answering, claims can be checked against the supplied source. In an open-domain task, there may be no single grounding document; external evidence or reference-free consistency checks may be needed. Consistency alone can miss a false claim that the model repeats reliably. section 5.1.1 section 5.5

Figure 5.3 The LLM Reliability Quadrant. Contextual grounding (vertical axis) asks whether the output stays within the provided sources. Factual alignment (horizontal axis) asks if the output is correct according to verifiable evidence. The safest output is both grounded and aligned. The riskiest output is neither grounded nor aligned. The mixed-signal quadrants are operationally important because the output can look based on source but wrong, or look correct but not grounded.

Use the reliability quadrant to distinguish source grounding from factual correctness: an unsupported claim can still happen to be true, and a grounded claim can still be wrong. section 5.1.2

In retrieval-augmented generation, faithfulness compares the answer with retrieved context. Pair it with answer relevance and retrieval measures: a faithful answer can still be incomplete if retrieval missed important information. Conversely, relevant retrieved documents do not guarantee that the answer uses them faithfully. section 6.1.2 section 6.3

Interpret the score alongside the claims and evidence. Ambiguous statements, context-dependent assertions, inference chains, and half-truths make both claim extraction and support judgments difficult. Define trusted sources and evaluation criteria explicitly, and use human safeguards for high-stakes decisions. section 1.1.3 section 5.1.2 section 5.4.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains claim decomposition, evidence verification, and score calculation, with caveats about ambiguous claims and support.
Examples
Calculates a 0.50 score for a financial answer with three supported claims and three unsupported claims.
What you can do after
Evaluate generated answers by decomposing claims, checking evidence, and interpreting faithfulness alongside retrieval quality.
How it connects
Distinguishes contextual grounding from factual alignment and relates faithfulness to answer relevance, context recall, and context precision.

Also explained in

Compare how 11 books explain fidelity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.