An answer can be accurate yet unhelpful if it addresses only part of the question. Answer relevance measures whether a generated response addresses the user’s question or information need. section 6.3.2
In retrieval-augmented generation (RAG), answer relevance compares the question with the response. It differs from retrieval relevance, which compares the question with retrieved context, and faithfulness, which checks whether the context supports the answer’s claims. A faithful answer can still miss the question. section 6.1.2 section 6.3.2
RAGAS estimates answer relevance using reverse question generation, a form of back-translation:
- Use a language model to generate several candidate questions from the answer. section 6.3.2
- Embed the original question and each candidate question as vectors. section 6.3.2
- Calculate the cosine similarity between the original question and each candidate, then average the similarities. section 6.3.2
An incomplete financial answer
An analyst asks about Q3 revenue drivers and Q4 risks. An accurate answer about Q3 alone may generate the candidate question “What drove Q3 revenue growth?” That candidate misses Q4 risks, so its similarity to the original question would be moderate, reflecting partial coverage. section 6.3.2
The score is reference-free: it uses the question and answer without a reference answer. This makes it useful for online monitoring, but it measures semantic alignment rather than whether claims are supported. Track it alongside faithfulness and retrieval metrics to distinguish response and retrieval failures. section 6.3.2 section 6.4.1
Table 6.4 Interpreting metric combinations for RAG diagnosis
|
Faithfulness
|
Answer relevance
|
Recall
|
Precision
|
Diagnosis
|
|
High
|
High
|
High
|
High
|
Working well
|
|
Low
|
High
|
High
|
High
|
Hallucination beyond retrieved context
|
|
High
|
Low
|
High
|
High
|
Off-topic generation. Misaligned.
|
|
High
|
High
|
Low
|
High
|
Retrieval missing relevant documents
|
|
High
|
High
|
Low
|
Low
|
Retrieval is fundamentally misaligned with the query
|
|
Low
|
Low
|
Low
|
Low
|
Systemic failure
|
Use the diagnostic matrix to compare answer relevance with faithfulness, recall, and precision before investigating a failure. section 6.4.1
Relevance can also mean something narrower in summarization: selecting important source content while avoiding redundancy and excess information. That criterion differs from measuring whether an answer addresses a question. section 4.2.5
















