liveBook · Manning

RAGGED

As explained in Retrieval Augmented Generation, The Foundational Ideas

RAGGED is an evaluation framework for testing retrieval-augmented generation assumptions across retrievers, reader models, datasets, and retrieval conditions.

RAGGED is an evaluation framework for testing retrieval-augmented generation assumptions across retrievers, reader models, datasets, and retrieval conditions. Its central finding is that reader robustness to irrelevant context can matter more than retriever quality at scale. section 15.0 section 15.1

The framework separates retrieval success from the reader’s ability to use retrieved evidence. With gold passages and no noise, even noise-sensitive models perform well; performance degrades when irrelevant passages accompany relevant ones. Thus, higher recall can deliver only modest answer-quality gains when the reader cannot filter noise. section 15.1.3

RAGGED supports two complementary stability questions:

  • RAG Stability Score (RSS) indicates whether small retrieval changes produce increasingly variable output quality. section 15.6.1 section 15.9
  • RAG Scalability Coefficient (RSC) characterizes how performance changes as retrieval depth increases, distinguishing peak-then-decline readers from improve-and-plateau readers. section 15.9
  • Reranking consistently improves both noise-robust and noise-sensitive readers, while prompting for focus has inconsistent effects and adds no benefit after reranking. section 15.1.3

Diagnosing a retrieval upgrade

A dense retriever can raise recall substantially without materially improving answer quality if the generator is overwhelmed by the additional documents. RAGGED therefore directs optimization toward reader noise tolerance, reranking, or context compression rather than assuming better embeddings will solve the failure. section 15.0 section 15.1.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Measures reader noise sensitivity, retrieval-depth behavior, and stability through RSS and RSC, then stops short of prescribing one architecture.
Examples
Uses gold-passage tests, Natural Questions and BioASQ comparisons, and a legal-document retrieval upgrade to expose reader bottlenecks.
What it assumes
Assumes familiarity with retrieval, readers, retrieved context, and RAG failure points rather than teaching those foundations from zero.
What you can do after
Evaluate RAG stability and retrieval-depth behavior, then select reranking or reader-focused interventions for measured failure points.
How it connects
Connects retrieval quality with reranking, context compression, Self-RAG, CRAG, RAGAS, and failure-taxonomy diagnosis.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.