Fluency measures the grammaticality, naturalness, and readability of generated language. It is evaluated alongside accuracy, coherence, and relevance to judge whether outputs suit real-world applications. section 12.1 section 12.6
Evaluate fluency as part of a broader, application-specific quality process:
- Test outputs on their intended tasks, such as summarization, question answering, or translation. section 12.1
- Combine automated metrics, manual review, user feedback, fair prompts, and ethical checks. section 12.1 section 9.3.2
- Compare results with baselines and monitor production performance to identify improvements or remaining weaknesses. section 9.3.2
Scoring summary fluency with G-Eval
A G-Eval workflow uses Azure OpenAI GPT-4 to score article summaries for relevance, coherence, consistency, and fluency. In the reported comparison, the LLM summary scored 3 for fluency and the NLP summary scored 2; the workflow stores metric results for comparison. section 12.3.2
Listing 12.2 LLM-based evaluation metrics
EVALUATION_PROMPT_TEMPLATE = """ #1
You will be given one summary written for an article. Your task [
is to rate the summary using one metric. Make sure you read
and understand these instructions very carefully.
Evaluation Criteria:
{criteria}
Evaluation Steps:
{steps}
Example:
Source Text:
{document}
Summary:
{summary}
Evaluation Form (scores ONLY):
- {metric_name}
"""
# Metric 1: Relevance #2
RELEVANCY_SCORE_CRITERIA = """
Relevance(1-5) - selection of important content from the source. \
The summary should include only important information from the
source document. \
Annotators were instructed to penalize summaries which contained
redundancies and excess information.
"""
RELEVANCY_SCORE_STEPS = """ #3
1. Read the summary and the source document carefully.
2. Compare the summary to the source document and identify the
main points of the article.
3. Assess how well the summary covers the main points of the article,
and how much irrelevant or redundant information it contains.
4. Assign a relevance score from 1 to 5.
"""
# Metric 2: Coherence #4
COHERENCE_SCORE_CRITERIA = """
Coherence(1-5) - the collective quality of all sentences. \
...
COHERENCE_SCORE_STEPS = """ #5
1. Read the article carefully and identify the main topic and key points.
2. Read the summary and compare it to the article. Check if the summary
...
# Metric 3: Consistency #6
CONSISTENCY_SCORE_CRITERIA = """
Consistency(1-5) - the factual alignment between the summary and the
summarized source.
...
CONSISTENCY_SCORE_STEPS = """ #7
1. Read the article carefully and identify its main facts and details.
2. Read the summary and compare it to the article. Check if the
summary ...
# Metric 4: Fluency #8
FLUENCY_SCORE_CRITERIA = """
Fluency(1-3): the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.
...
FLUENCY_SCORE_STEPS = """ #9
Read the summary and evaluate its fluency based on the given criteria.
Assign a fluency score from 1 to 3.
...
Inspect the evaluation prompt template and metric criteria used to turn fluency into a repeatable summary-scoring task. section 12.3.2
