eRAG addresses the correlation gap between healthy retrieval metrics and poor generated answers. Traditional labels reward documents that match a query topic, even when the downstream language model cannot use them to answer correctly. section 14.1.2
For each retrieved document, eRAG asks the consuming language model to answer using that document alone, then scores the answer against ground truth with task-specific measures such as Exact Match or F1. Retrieval quality therefore reflects usefulness to generation rather than relevance alone. section 14.1.2
The method changes both the evaluation target and the evaluation workload:
- Replace the question “Is this document relevant?” with “Does this document help the LLM produce the correct answer?” section 14.1.2
- Evaluate documents independently instead of placing the entire retrieval list in one model context, reducing memory consumption by 30–48× compared with end-to-end evaluation. section 14.1.2
- Use correlation with downstream RAG performance as the test of an evaluation method; eRAG outperformed traditional approaches across four KILT datasets, with Kendall’s tau improvements of 0.168–0.494. section 14.1.2 section 14.7
Per-document evaluation
Given a query, retrieved document, and reference answer, generate one answer from that document and score it with Exact Match or F1. Repeat this independently for every retrieved document, then compare the resulting judgments with actual RAG performance. section 14.1.2
Figure 14.2 Traditional vs LLM-based retrieval evaluation.

Compare binary human relevance labels and aggregate retrieval metrics with per-document LLM judgments based on generated-answer quality. section 14.1.2
