Answer relevancy measures whether an agent directly addresses the core intent of a user’s question rather than evading, answering a different question, or replacing the answer with background. section 10.2.4 section 10.3.2
Apply answer relevancy as a binary rubric item:
- PASS when the response directly answers the user’s core question and provides a clear takeaway or conclusion. section 10.2.4 section 10.3.2
- FAIL when the response gives only a plausible summary, answers another question, reframes the request, rambles with background, or lacks a conclusion. section 10.2.4 section 10.3.2
- Prefer PASS/FAIL decisions when consistency matters; graded scales require harder-to-stabilize distinctions between adjacent scores. section 10.2.4
A rubric-based evaluator receives the user query, final answer, rubric, and, when available, an expected answer. For research agents, it can also receive an evidence pack containing source identifiers, metadata, and relevant excerpts. The evaluator must judge only the supplied evidence, avoid rewarding verbosity or friendliness, and return one JSON object. Track the answer relevancy rate as the fraction of evaluations passing this rubric item; combine it with citation reliability and requirement compliance in an overall pass rate. section 10.3.2 section 10.2.4
