Ragas is an open-source framework for diagnosing retrieval-augmented generation (RAG) systems across retrieval and generation. It separates failures caused by missing or noisy context from failures caused by unsupported or irrelevant answers. section 6.0 section 6.3
Table 6.3 The RAGAS evaluation matrix
|
Component
|
Metric
|
Question asked
|
|
Generation
|
Faithfulness
|
Are the answer’s claims supported by the retrieved context?
|
|
Generation
|
Answer relevance
|
Does the answer actually address the user’s question?
|
|
Retrieval
|
Context recall
|
Does the retrieved context contain the information needed to answer?
|
|
Retrieval
|
Context precision
|
Is the retrieved context free of noise and irrelevant content?
|
Use the evaluation matrix to map each metric to the RAG component and question it evaluates. section 6.3
Evaluate four complementary dimensions:
- Faithfulness checks whether the answer's claims are supported by retrieved context, using claim decomposition. section 6.3 section 6.5
- Answer relevance checks whether the response addresses the user's question, using back-translation. section 6.3 section 6.5
- Context recall checks whether retrieved context contains the information needed for the ground-truth answer. section 6.3 section 6.5
- Context precision checks whether relevant passages appear above irrelevant content and noise. section 6.3 section 6.5
Diagnosing a financial research assistant
A retrieved earnings report supports claims about $2.3 billion in revenue, 15% year-over-year growth, and cloud services as the growth driver. If the answer also invents European expansion and a new CEO, faithfulness is low despite relevant context. High faithfulness with low context recall instead indicates that retrieval omitted information required by the answer. section 6.0 section 6.5



