Coherence measures whether generated content remains logically consistent, with ideas transitioning smoothly from one sentence to the next. It is a qualitative evaluation metric for judging whether outputs are understandable and meaningful in context. section 12.3.2 section 9.3.2
Use coherence evaluation to:
- Assess summaries, question-answering outputs, translations, and other application-specific NLP results. section 12.1
- Combine logical-flow judgments with relevance, consistency, fluency, accuracy, and user-experience measures. section 12.1 section 9.3.2
- Track model quality during benchmarking, fine-tuning, and continuous improvement rather than treating one score as production readiness. section 12.1 section 12.6
G-Eval summary comparison
Compare two summaries of a Gates Foundation AI-principles article: one produced by an NLP library and one by Gemini Pro 1.5. A G-Eval workflow uses an Azure OpenAI GPT-4 model to score relevance, coherence, consistency, and fluency from 1 to 5. The displayed results give the LLM summary a coherence score of 5 and the NLP summary a score of 1. section 12.3.2
Listing 12.2 LLM-based evaluation metrics
EVALUATION_PROMPT_TEMPLATE = """ #1
You will be given one summary written for an article. Your task [
is to rate the summary using one metric. Make sure you read
and understand these instructions very carefully.
Evaluation Criteria:
{criteria}
Evaluation Steps:
{steps}
Example:
Source Text:
{document}
Summary:
{summary}
Evaluation Form (scores ONLY):
- {metric_name}
"""
# Metric 1: Relevance #2
RELEVANCY_SCORE_CRITERIA = """
Relevance(1-5) - selection of important content from the source. \
The summary should include only important information from the
source document. \
Annotators were instructed to penalize summaries which contained
redundancies and excess information.
"""
RELEVANCY_SCORE_STEPS = """ #3
1. Read the summary and the source document carefully.
2. Compare the summary to the source document and identify the
main points of the article.
3. Assess how well the summary covers the main points of the article,
and how much irrelevant or redundant information it contains.
4. Assign a relevance score from 1 to 5.
"""
# Metric 2: Coherence #4
COHERENCE_SCORE_CRITERIA = """
Coherence(1-5) - the collective quality of all sentences. \
...
COHERENCE_SCORE_STEPS = """ #5
1. Read the article carefully and identify the main topic and key points.
2. Read the summary and compare it to the article. Check if the summary
...
# Metric 3: Consistency #6
CONSISTENCY_SCORE_CRITERIA = """
Consistency(1-5) - the factual alignment between the summary and the
summarized source.
...
CONSISTENCY_SCORE_STEPS = """ #7
1. Read the article carefully and identify its main facts and details.
2. Read the summary and compare it to the article. Check if the
summary ...
# Metric 4: Fluency #8
FLUENCY_SCORE_CRITERIA = """
Fluency(1-3): the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.
...
FLUENCY_SCORE_STEPS = """ #9
Read the summary and evaluate its fluency based on the given criteria.
Assign a fluency score from 1 to 3.
...
Inspect the evaluation prompt template and metric criteria used to request structured scores for a summary against its source document. section 12.3.2
