liveBook · Manning

Evaluation triad

As explained in Retrieval Augmented Generation, The Foundational Ideas

The evaluation triad is a three-dimensional evaluation framework comprising faithfulness, answer relevance, and context relevance.

The evaluation triad evaluates retrieval-augmented generation through three independent relationships among the query, retrieved context, and generated response: faithfulness, answer relevance, and context relevance. It requires no ground-truth answers, so it can assess previously unseen queries. section 14.2

Use each dimension to isolate a different failure point:

  • Context relevance detects retrieval problems by asking whether the retrieved material helps answer the query. section 14.2
  • Faithfulness detects generation hallucinations by testing whether the response stays supported by the retrieved context. section 14.2
  • Answer relevance detects extraction and specificity failures by testing whether the response addresses the query. section 14.2

A reranker regression

A mortgage compliance assistant improved retrieval after adding a cross-encoder reranker, but faithfulness fell as answers drifted from their supporting documents. Comparing the triad across releases exposes this regression instead of relying on a single threshold. section 14.0

Figure 14.3 The RAGAS evaluation triad.

The triad diagram maps each metric to one relationship among the query, retrieved context, and generated response. section 14.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Decomposes quality into three reference-free metrics and connects each metric to a pipeline failure, without detailing scoring internals.
Examples
Uses a mortgage compliance assistant whose reranker improved retrieval but reduced faithfulness.
What it assumes
Assumes a RAG pipeline producing a query, retrieved context, and generated response.
What you can do after
Diagnoses whether RAG failures originate in retrieval, generation, or their interaction without maintaining curated answers.
How it connects
Connects retrieval, generation, hallucination, answer relevance, context relevance, faithfulness, and LLM-as-judge evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.