liveBook · Manning

Fidelity

As explained in Evaluating AI Systems

A measure of whether a response is faithful to its source information or supporting evidence.

Fidelity, or faithfulness, measures whether a generated response is supported by its source information. In a retrieval-augmented generation (RAG) system, it assesses whether an answer is grounded in the retrieved evidence rather than containing unsupported claims. section 3.2

A RAG faithfulness evaluation has four elements:

  • Inputs: the generated answer and its retrieved context. section 3.2
  • Computation: automated comparison, such as semantic similarity or entailment modeling, or human or language-model judgment of statement support. section 3.2
  • Output: a score indicating the proportion or degree of factual consistency between answer and context. section 3.2
  • Interpretation: a high score indicates grounding in retrieved sources, but does not establish that the answer is complete, useful, or well-formulated. section 3.2
Table 3.1 Overview of common AI evaluation metrics.
Metric
Input
Computation
Output
Interpretation
RAG faithfulness
The system’s generated answer and the retrieved context that supports it
Automated comparison techniques such as semantic similarity or entailment modeling, or judgment-based evaluations where humans or language models assess whether each statement in the answer is supported by retrieved context
A score indicating the proportion or degree of factual consistency between the generated answer and the context.
High faithfulness means that the system grounds its outputs in the retrieved sources rather than hallucinating unsupported content. It does not necessarily mean the answer is complete, useful, or well-formulated;
Agent task success rate
Logs of the agent’s actions, goals, and outcomes over multiple task runs
Calculate the proportion of tasks in which the agent successfully achieved the intended goal without human intervention or unintended side effects.
Success ratio or percentage, sometimes complemented by qualitative notes about the nature of failures
High success rate means that the systems is reliably achieving its assigned objectives. It cannot say whether an agent has taken any unsafe shortcuts or violated constraints in order to achieve the task
Word Error Rate for Automatic Speech Recognition (ASR)
The ASR system’s generated text and a reference text
The minimum number of word-level insertions, deletions, and substitutions required to transform the generated text into the reference text, divided by the total number of words in the reference.
A positive real number, with 0 as its minimum value
A lower WER indicates closer surface-level similarity between the generated text and the reference.  It does not capture semantic adequacy, fluency, or usefulness.

Compare the inputs, computation, output, and interpretation of RAG faithfulness with those of other evaluation metrics. section 3.2

How Evaluating AI Systems covers it

How deep it goes
Defines the metric and outlines its inputs, scoring approaches, output, and limitations without specifying a single implementation.
Examples
Uses a generated RAG answer and its retrieved context as evaluation inputs.
What you can do after
Evaluate whether an answer is supported by retrieved context while recognizing that faithfulness alone does not establish completeness or usefulness.
How it connects
Connects faithfulness with retrieval-augmented generation, semantic similarity, entailment modeling, and hallucination.

Also explained in

Compare how 11 books explain fidelity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.