liveBook · Manning

eRAG

As explained in Retrieval Augmented Generation, The Foundational Ideas

eRAG is a retrieval-augmented generation evaluation method that uses per-document language-model judgments to estimate actual system performance.

eRAG addresses the correlation gap between healthy retrieval metrics and poor generated answers. Traditional labels reward documents that match a query topic, even when the downstream language model cannot use them to answer correctly. section 14.1.2

For each retrieved document, eRAG asks the consuming language model to answer using that document alone, then scores the answer against ground truth with task-specific measures such as Exact Match or F1. Retrieval quality therefore reflects usefulness to generation rather than relevance alone. section 14.1.2

The method changes both the evaluation target and the evaluation workload:

  • Replace the question “Is this document relevant?” with “Does this document help the LLM produce the correct answer?” section 14.1.2
  • Evaluate documents independently instead of placing the entire retrieval list in one model context, reducing memory consumption by 30–48× compared with end-to-end evaluation. section 14.1.2
  • Use correlation with downstream RAG performance as the test of an evaluation method; eRAG outperformed traditional approaches across four KILT datasets, with Kendall’s tau improvements of 0.168–0.494. section 14.1.2 section 14.7

Per-document evaluation

Given a query, retrieved document, and reference answer, generate one answer from that document and score it with Exact Match or F1. Repeat this independently for every retrieved document, then compare the resulting judgments with actual RAG performance. section 14.1.2

Figure 14.2 Traditional vs LLM-based retrieval evaluation.

Compare binary human relevance labels and aggregate retrieval metrics with per-document LLM judgments based on generated-answer quality. section 14.1.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines per-document judging, answer scoring, benchmark correlation, and memory savings, but does not specify implementation code.
Examples
Uses Exact Match or F1 scoring across KILT datasets including Natural Questions, TriviaQA, HotpotQA, and FEVER.
What it assumes
Assumes a downstream LLM, retrieved documents, ground-truth answers, and a task-specific scoring metric.
What you can do after
Evaluate retrieval with per-document LLM judgments and task-specific answer metrics.
How it connects
Connects retrieval metrics, relevance labels, language-model evaluation, ground truth, and downstream RAG performance.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.