liveBook · Manning

Reference-free metric

As explained in Evaluating AI Systems

A metric that assesses system outputs without comparing them to a fixed ground truth.

A reference-free metric assesses a system’s output without comparing it to a fixed ground truth. It is useful when a correct answer does not exist or is too expensive to produce. Unlike a reference-based metric, it can assess an output’s properties, relationship to its input or context, or compliance with constraints. section 3.2

Some reference-free metrics are deterministic, such as latency, output length, format compliance, or an agent’s number of steps. Others use models or judges to assess qualities such as faithfulness, toxicity, safety, coherence, or fluency. section 3.2

Faithfulness to retrieved context

A faithfulness metric assesses whether a generated answer is supported by retrieved evidence. A high score indicates grounding in that context, not necessarily that the answer is complete, useful, or well-formulated. section 3.2

Table 3.1 Overview of common AI evaluation metrics.
Metric
Input
Computation
Output
Interpretation
RAG faithfulness
The system’s generated answer and the retrieved context that supports it
Automated comparison techniques such as semantic similarity or entailment modeling, or judgment-based evaluations where humans or language models assess whether each statement in the answer is supported by retrieved context
A score indicating the proportion or degree of factual consistency between the generated answer and the context.
High faithfulness means that the system grounds its outputs in the retrieved sources rather than hallucinating unsupported content. It does not necessarily mean the answer is complete, useful, or well-formulated;
Agent task success rate
Logs of the agent’s actions, goals, and outcomes over multiple task runs
Calculate the proportion of tasks in which the agent successfully achieved the intended goal without human intervention or unintended side effects.
Success ratio or percentage, sometimes complemented by qualitative notes about the nature of failures
High success rate means that the systems is reliably achieving its assigned objectives. It cannot say whether an agent has taken any unsafe shortcuts or violated constraints in order to achieve the task
Word Error Rate for Automatic Speech Recognition (ASR)
The ASR system’s generated text and a reference text
The minimum number of word-level insertions, deletions, and substitutions required to transform the generated text into the reference text, divided by the total number of words in the reference.
A positive real number, with 0 as its minimum value
A lower WER indicates closer surface-level similarity between the generated text and the reference.  It does not capture semantic adequacy, fluency, or usefulness.

The metric overview illustrates how faithfulness is computed and how its score should be interpreted. section 3.2

How Evaluating AI Systems covers it

How deep it goes
Defines the category and contrasts deterministic and judgment-based examples, without a detailed implementation procedure.
Examples
Illustrates the category with latency, format compliance, agent step counts, faithfulness, toxicity, and fluency.
What you can do after
Evaluate and understand reference-free signals while recognizing that faithfulness alone does not establish completeness or usefulness.
How it connects
Connects reference-free metrics to reference-based metrics, faithfulness, and retrieval-augmented generation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.