A reference-free metric assesses a system’s output without comparing it to a fixed ground truth. It is useful when a correct answer does not exist or is too expensive to produce. Unlike a reference-based metric, it can assess an output’s properties, relationship to its input or context, or compliance with constraints. section 3.2
Some reference-free metrics are deterministic, such as latency, output length, format compliance, or an agent’s number of steps. Others use models or judges to assess qualities such as faithfulness, toxicity, safety, coherence, or fluency. section 3.2
Faithfulness to retrieved context
A faithfulness metric assesses whether a generated answer is supported by retrieved evidence. A high score indicates grounding in that context, not necessarily that the answer is complete, useful, or well-formulated. section 3.2
Table 3.1 Overview of common AI evaluation metrics.
|
Metric
|
Input
|
Computation
|
Output
|
Interpretation
|
|
RAG faithfulness
|
The system’s generated answer and the retrieved context that supports it
|
Automated comparison techniques such as semantic similarity or entailment modeling, or judgment-based evaluations where humans or language models assess whether each statement in the answer is supported by retrieved context
|
A score indicating the proportion or degree of factual consistency between the generated answer and the context.
|
High faithfulness means that the system grounds its outputs in the retrieved sources rather than hallucinating unsupported content. It does not necessarily mean the answer is complete, useful, or well-formulated;
|
|
Agent task success rate
|
Logs of the agent’s actions, goals, and outcomes over multiple task runs
|
Calculate the proportion of tasks in which the agent successfully achieved the intended goal without human intervention or unintended side effects.
|
Success ratio or percentage, sometimes complemented by qualitative notes about the nature of failures
|
High success rate means that the systems is reliably achieving its assigned objectives. It cannot say whether an agent has taken any unsafe shortcuts or violated constraints in order to achieve the task
|
|
Word Error Rate for Automatic Speech Recognition (ASR)
|
The ASR system’s generated text and a reference text
|
The minimum number of word-level insertions, deletions, and substitutions required to transform the generated text into the reference text, divided by the total number of words in the reference.
|
A positive real number, with 0 as its minimum value
|
A lower WER indicates closer surface-level similarity between the generated text and the reference. It does not capture semantic adequacy, fluency, or usefulness.
|
The metric overview illustrates how faithfulness is computed and how its score should be interpreted. section 3.2
