Answer relevance measures how directly a generated response addresses the user’s query and intended level of detail. A response can rely on correct retrieved documents yet remain irrelevant if it answers only part of the request or uses the wrong format. section 7.3.2
Evaluate relevance by checking whether the response:
- Addresses the specific request rather than merely discussing a related topic. section 7.3.2
- Matches the requested depth, such as detailed analysis, statistics, or case studies. section 7.3.2
- Fits the user’s requested style or transformation when those requirements are part of the task. section 5.1.3
A relevant report
For a request for a detailed report on AI trends in SaaS, a short blog-style overview that only skims the topic has low answer relevance, even when its claims come from correct documents. A useful evaluation asks whether the response supplies the depth and content the query demands. section 7.3.2
Answer relevance is one part of RAG evaluation alongside context relevance and groundedness. Teams can use LLM-as-a-Judge assessments, then refine prompts, retrieval, models, or criteria as user feedback reveals gaps. Judge quality depends heavily on the selected evaluation criterion. section 7.3.2 section 7.4
