Relevance measures whether generated content stays aligned with the input’s context and intent, answers the task directly, and avoids unnecessary information. It helps teams judge whether an LLM output is useful for applications such as question answering, summarization, chatbots, and fine-tuned workflows. section 12.1 section 9.3.2
Evaluate relevance by checking:
- Whether the output selects important content from the source rather than adding excess or redundant information. section 12.3.2
- Whether the response stays on topic and provides information directly applicable to the query or task. section 9.3.2
- Whether the metric is applied alongside coherence, consistency, fluency, accuracy, user feedback, and ethical checks rather than used alone. section 12.1 section 9.3.2
G-Eval summary comparison
Compare two summaries of a Bill and Melinda Gates Foundation AI-principles article using an Azure OpenAI GPT-4 evaluator. A five-point relevance rubric scores selection of important source content and penalizes redundancy; the reported LLM summary scores 5 while the NLP summary scores 2. section 12.3.2
Listing 12.2 LLM-based evaluation metrics
EVALUATION_PROMPT_TEMPLATE = """ #1
You will be given one summary written for an article. Your task [
is to rate the summary using one metric. Make sure you read
and understand these instructions very carefully.
Evaluation Criteria:
{criteria}
Evaluation Steps:
{steps}
Example:
Source Text:
{document}
Summary:
{summary}
Evaluation Form (scores ONLY):
- {metric_name}
"""
# Metric 1: Relevance #2
RELEVANCY_SCORE_CRITERIA = """
Relevance(1-5) - selection of important content from the source. \
The summary should include only important information from the
source document. \
Annotators were instructed to penalize summaries which contained
redundancies and excess information.
"""
RELEVANCY_SCORE_STEPS = """ #3
1. Read the summary and the source document carefully.
2. Compare the summary to the source document and identify the
main points of the article.
3. Assess how well the summary covers the main points of the article,
and how much irrelevant or redundant information it contains.
4. Assign a relevance score from 1 to 5.
"""
# Metric 2: Coherence #4
COHERENCE_SCORE_CRITERIA = """
Coherence(1-5) - the collective quality of all sentences. \
...
COHERENCE_SCORE_STEPS = """ #5
1. Read the article carefully and identify the main topic and key points.
2. Read the summary and compare it to the article. Check if the summary
...
# Metric 3: Consistency #6
CONSISTENCY_SCORE_CRITERIA = """
Consistency(1-5) - the factual alignment between the summary and the
summarized source.
...
CONSISTENCY_SCORE_STEPS = """ #7
1. Read the article carefully and identify its main facts and details.
2. Read the summary and compare it to the article. Check if the
summary ...
# Metric 4: Fluency #8
FLUENCY_SCORE_CRITERIA = """
Fluency(1-3): the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.
...
FLUENCY_SCORE_STEPS = """ #9
Read the summary and evaluate its fluency based on the given criteria.
Assign a fluency score from 1 to 3.
...
Inspect the reusable evaluation prompt, relevance criteria, and scoring steps used to operationalize the metric. section 12.3.2
Figure 12.2 DeepEval test session example
View a DeepEval test-session example for applying answer relevancy within an LLM evaluation workflow. section 12.3.8
