Fidelity, or faithfulness, measures whether a generated response is supported by its source information. In a retrieval-augmented generation (RAG) system, it assesses whether an answer is grounded in the retrieved evidence rather than containing unsupported claims. section 3.2
A RAG faithfulness evaluation has four elements:
- Inputs: the generated answer and its retrieved context. section 3.2
- Computation: automated comparison, such as semantic similarity or entailment modeling, or human or language-model judgment of statement support. section 3.2
- Output: a score indicating the proportion or degree of factual consistency between answer and context. section 3.2
- Interpretation: a high score indicates grounding in retrieved sources, but does not establish that the answer is complete, useful, or well-formulated. section 3.2
Table 3.1 Overview of common AI evaluation metrics.
|
Metric
|
Input
|
Computation
|
Output
|
Interpretation
|
|
RAG faithfulness
|
The system’s generated answer and the retrieved context that supports it
|
Automated comparison techniques such as semantic similarity or entailment modeling, or judgment-based evaluations where humans or language models assess whether each statement in the answer is supported by retrieved context
|
A score indicating the proportion or degree of factual consistency between the generated answer and the context.
|
High faithfulness means that the system grounds its outputs in the retrieved sources rather than hallucinating unsupported content. It does not necessarily mean the answer is complete, useful, or well-formulated;
|
|
Agent task success rate
|
Logs of the agent’s actions, goals, and outcomes over multiple task runs
|
Calculate the proportion of tasks in which the agent successfully achieved the intended goal without human intervention or unintended side effects.
|
Success ratio or percentage, sometimes complemented by qualitative notes about the nature of failures
|
High success rate means that the systems is reliably achieving its assigned objectives. It cannot say whether an agent has taken any unsafe shortcuts or violated constraints in order to achieve the task
|
|
Word Error Rate for Automatic Speech Recognition (ASR)
|
The ASR system’s generated text and a reference text
|
The minimum number of word-level insertions, deletions, and substitutions required to transform the generated text into the reference text, divided by the total number of words in the reference.
|
A positive real number, with 0 as its minimum value
|
A lower WER indicates closer surface-level similarity between the generated text and the reference. It does not capture semantic adequacy, fluency, or usefulness.
|
Compare the inputs, computation, output, and interpretation of RAG faithfulness with those of other evaluation metrics. section 3.2










