liveBook · Manning

Relevance

As explained in Build an Advanced RAG Application (From Scratch)

Answer relevancy measures how closely a generated answer addresses the original question, including through similarity to questions generated from the answer.

Answer relevancy measures whether a generated response addresses the user's original question instead of drifting into generic or tangential information. It evaluates the generation component of a RAG pipeline and complements retrieval-focused measures such as context relevance. section 6.5.1 section 6.5

RAGAS estimates answer relevancy by generating several artificial questions from the answer, then measuring their similarity to the original question. This goes beyond simple keyword matching by testing whether the answer expresses the question's intended focus. section 6.5.2

Interpret the metric alongside neighboring RAG checks:

  • Compare it with faithfulness, which checks whether answer claims follow from retrieved context rather than whether the answer addresses the question. section 6.5.1
  • Pair it with context relevance and context recall to distinguish a poorly targeted answer from missing or insufficient retrieval context. section 6.5.1 section 6.5.2
  • Use it as one component of the composite RAGAS score when comparing configurations or tracking system improvements. section 6.5.2
Figure 6.7 RAGAS Core Metric extensively covers LLM Grounding and Response (Source)

Use the RAGAS metric figure to see how answer relevancy fits alongside grounding and response evaluation. section 6.5.2

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines the metric and its artificial-question similarity procedure, but does not provide implementation code or scoring details.
Examples
Generates artificial questions from an answer and compares them with the original question rather than relying on keyword matching.
What it assumes
Assumes familiarity with RAG retrieval and generation, while distinguishing answer targeting from context relevance and faithfulness.
What you can do after
Evaluate RAG answer targeting and compare system configurations using answer relevancy within the RAGAS metric suite.
How it connects
Connects answer relevancy with RAGAS, faithfulness, context relevance, and context recall.

Also explained in

Compare how 17 books explain relevance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.