liveBook · Manning

Ragas

As explained in LLM Evaluation and Alignment, The Foundational Ideas

RAGAS is an open-source framework for evaluating retrieval-augmented generation systems with metrics such as faithfulness.

Ragas is an open-source framework for diagnosing retrieval-augmented generation (RAG) systems across retrieval and generation. It separates failures caused by missing or noisy context from failures caused by unsupported or irrelevant answers. section 6.0 section 6.3

Table 6.3 The RAGAS evaluation matrix
Component
Metric
Question asked
Generation
Faithfulness
Are the answer’s claims supported by the retrieved context?
Generation
Answer relevance
Does the answer actually address the user’s question?
Retrieval
Context recall
Does the retrieved context contain the information needed to answer?
Retrieval
Context precision
Is the retrieved context free of noise and irrelevant content?

Use the evaluation matrix to map each metric to the RAG component and question it evaluates. section 6.3

Evaluate four complementary dimensions:

  • Faithfulness checks whether the answer's claims are supported by retrieved context, using claim decomposition. section 6.3 section 6.5
  • Answer relevance checks whether the response addresses the user's question, using back-translation. section 6.3 section 6.5
  • Context recall checks whether retrieved context contains the information needed for the ground-truth answer. section 6.3 section 6.5
  • Context precision checks whether relevant passages appear above irrelevant content and noise. section 6.3 section 6.5

Diagnosing a financial research assistant

A retrieved earnings report supports claims about $2.3 billion in revenue, 15% year-over-year growth, and cloud services as the growth driver. If the answer also invents European expansion and a new CEO, faithfulness is low despite relevant context. High faithfulness with low context recall instead indicates that retrieval omitted information required by the answer. section 6.0 section 6.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines four metrics and their diagnostic combinations, while deferring each metric's full measurement methodology.
Examples
Uses a financial research assistant, earnings-report claims, hallucinated business details, and high-low metric combinations.
What it assumes
Assumes a RAG system with retrieval and generation components, while distinguishing their responsibilities explicitly.
What you can do after
Diagnose/debug with faithfulness, answer relevance, context recall, and context precision before selecting retrieval or generation fixes.
How it connects
Connects retrieval evaluation with information retrieval, hallucination detection, claim decomposition, and answer relevance.

Also explained in

Compare how 4 books explain Ragas →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.