liveBook · Manning

CRAG

As explained in Retrieval Augmented Generation, The Foundational Ideas

The Comprehensive RAG Benchmark is a benchmark for evaluating retrieval-augmented generation systems on question-answering performance.

CRAG is the Comprehensive RAG Benchmark, used to evaluate question-answering accuracy for large language models and retrieval-augmented generation (RAG) systems. It is distinct from Corrective Retrieval-Augmented Generation, a retrieval-auditing approach that shares the acronym. section 11.1.3

The reported results show why adding retrieval is not enough on its own: most advanced LLMs reached at most 34% accuracy on the benchmark, while straightforward RAG raised accuracy only to 44%. These figures describe benchmark performance, not the effect of the corrective approach. section 11.1.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Reports headline accuracy results without detailing benchmark tasks or scoring procedures.
Examples
Compares at-most-34% accuracy for most advanced LLMs with 44% for straightforward RAG.
What you can do after
Evaluate/understand the reported limits of straightforward RAG without attributing those results to the corrective approach.
How it connects
Distinguishes the Comprehensive RAG Benchmark from Corrective Retrieval-Augmented Generation while relating its results to LLMs and RAG.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.