liveBook · Manning

Relevance

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Answer relevance is a generation-quality metric measuring how directly and appropriately a generated answer addresses the question or information need.

An answer can be accurate yet unhelpful if it addresses only part of the question. Answer relevance measures whether a generated response addresses the user’s question or information need. section 6.3.2

In retrieval-augmented generation (RAG), answer relevance compares the question with the response. It differs from retrieval relevance, which compares the question with retrieved context, and faithfulness, which checks whether the context supports the answer’s claims. A faithful answer can still miss the question. section 6.1.2 section 6.3.2

RAGAS estimates answer relevance using reverse question generation, a form of back-translation:

  • Use a language model to generate several candidate questions from the answer. section 6.3.2
  • Embed the original question and each candidate question as vectors. section 6.3.2
  • Calculate the cosine similarity between the original question and each candidate, then average the similarities. section 6.3.2

An incomplete financial answer

An analyst asks about Q3 revenue drivers and Q4 risks. An accurate answer about Q3 alone may generate the candidate question “What drove Q3 revenue growth?” That candidate misses Q4 risks, so its similarity to the original question would be moderate, reflecting partial coverage. section 6.3.2

The score is reference-free: it uses the question and answer without a reference answer. This makes it useful for online monitoring, but it measures semantic alignment rather than whether claims are supported. Track it alongside faithfulness and retrieval metrics to distinguish response and retrieval failures. section 6.3.2 section 6.4.1

Table 6.4 Interpreting metric combinations for RAG diagnosis
Faithfulness
Answer relevance
Recall
Precision
Diagnosis
High
High
High
High
Working well
Low
High
High
High
Hallucination beyond retrieved context
High
Low
High
High
Off-topic generation. Misaligned.
High
High
Low
High
Retrieval missing relevant documents
High
High
Low
Low
Retrieval is fundamentally misaligned with the query
Low
Low
Low
Low
Systemic failure

Use the diagnostic matrix to compare answer relevance with faithfulness, recall, and precision before investigating a failure. section 6.4.1

Relevance can also mean something narrower in summarization: selecting important source content while avoiding redundancy and excess information. That criterion differs from measuring whether an answer addresses a question. section 4.2.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines a three-step back-translation procedure using generated questions, embeddings, and average cosine similarity, with a worked partial-answer example.
Examples
Uses a financial research question about Q3 revenue drivers and Q4 risks to illustrate an accurate but incomplete answer.
What you can do after
Diagnose RAG response failures by comparing answer relevance with faithfulness and retrieval scores.
How it connects
Distinguishes answer relevance from faithfulness and retrieval relevance, and combines it with context recall and context precision for diagnosis.

Also explained in

Compare how 17 books explain relevance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.