G-Eval uses an LLM to judge generated text against specified quality criteria. It is reference-free: a summary can be evaluated with its source article without needing a prewritten ideal summary. This is useful when content is new or tailored to individual cases, making reference answers scarce. section 12.3.1
The evaluator receives the text, a criterion, instructions for applying it, and an evaluation form. Criteria can address coherence, consistency, relevance, or fluency. Prompted evaluation steps guide the judgment; the form requests a score rather than an unrestricted response. section 12.3.1 section 12.3.2
To compare summaries:
- Define each criterion, its scoring scale, and the steps for applying it. section 12.3.2
- Provide the source article and one summary to the evaluator for each criterion. section 12.3.2
- Collect the scores by summary and criterion, then compare the results rather than relying on a single overall impression. section 12.3.2
Comparing two article summaries
An Azure OpenAI GPT-4 evaluator scores an NLP-generated summary and an LLM-generated summary of an article about AI principles. The LLM summary scores 5 for coherence, consistency, and relevance, versus 1, 4, and 2 for the NLP summary; their fluency scores are 3 and 2. These are judgments under the chosen prompts and criteria, not a substitute for human evaluation. section 12.3.2 section 12.6
Listing 12.2 LLM-based evaluation metrics
EVALUATION_PROMPT_TEMPLATE = """ #1
You will be given one summary written for an article. Your task [
is to rate the summary using one metric. Make sure you read
and understand these instructions very carefully.
Evaluation Criteria:
{criteria}
Evaluation Steps:
{steps}
Example:
Source Text:
{document}
Summary:
{summary}
Evaluation Form (scores ONLY):
- {metric_name}
"""
# Metric 1: Relevance #2
RELEVANCY_SCORE_CRITERIA = """
Relevance(1-5) - selection of important content from the source. \
The summary should include only important information from the
source document. \
Annotators were instructed to penalize summaries which contained
redundancies and excess information.
"""
RELEVANCY_SCORE_STEPS = """ #3
1. Read the summary and the source document carefully.
2. Compare the summary to the source document and identify the
main points of the article.
3. Assess how well the summary covers the main points of the article,
and how much irrelevant or redundant information it contains.
4. Assign a relevance score from 1 to 5.
"""
# Metric 2: Coherence #4
COHERENCE_SCORE_CRITERIA = """
Coherence(1-5) - the collective quality of all sentences. \
...
COHERENCE_SCORE_STEPS = """ #5
1. Read the article carefully and identify the main topic and key points.
2. Read the summary and compare it to the article. Check if the summary
...
# Metric 3: Consistency #6
CONSISTENCY_SCORE_CRITERIA = """
Consistency(1-5) - the factual alignment between the summary and the
summarized source.
...
CONSISTENCY_SCORE_STEPS = """ #7
1. Read the article carefully and identify its main facts and details.
2. Read the summary and compare it to the article. Check if the
summary ...
# Metric 4: Fluency #8
FLUENCY_SCORE_CRITERIA = """
Fluency(1-3): the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.
...
FLUENCY_SCORE_STEPS = """ #9
Read the summary and evaluate its fluency based on the given criteria.
Assign a fluency score from 1 to 3.
...
Inspect the prompt template to see how the source, summary, criterion, evaluation steps, and score-only form fit together. section 12.3.2
Listing 12.3 LLM-based evaluation metrics
def get_article(): #1
with open('./data/gates_foundation_article.txt', 'r') as f:
article = f.read()
with open('./data/gates_foundation_summary1.txt', 'r') as f:
nlp_summary = f.read()
with open('./data/gates_foundation_summary2.txt', 'r') as f:
llm_summary = f.read()
return article, nlp_summary, llm_summary
def get_geval_score(criteria: str, steps: str, document: str,
summary: str, metric_name: str): #2
prompt = EVALUATION_PROMPT_TEMPLATE.format( #3
criteria=criteria,
steps=steps,
metric_name=metric_name,
document=document,
summary=summary,
)
response = client.chat.completions.create( #4
model=MODEL,
messages = [{"role": "user", "content": prompt}],
temperature = 0,
max_tokens = 5,
top_p = 1,
frequency_penalty = 0,
presence_penalty = 0,
stop = None
)
return response.choices[0].message.content
evaluation_metrics = {
"Relevance": (RELEVANCY_SCORE_CRITERIA, RELEVANCY_SCORE_STEPS),
"Coherence": (COHERENCE_SCORE_CRITERIA, COHERENCE_SCORE_STEPS),
"Consistency": (CONSISTENCY_SCORE_CRITERIA, CONSISTENCY_SCORE_STEPS),
"Fluency": (FLUENCY_SCORE_CRITERIA, FLUENCY_SCORE_STEPS),
}
# Main code
client = AzureOpenAI(
azure_endpoint = AZURE_ENDPOINT,
api_key=AOAI_API_KEY,
api_version=API_VERSION
)
article_text, nlp_summary, llm_summary = get_article()
summaries = {"NLP Summary (1)": nlp_summary,
"LLM Summary (2)": llm_summary}
data = {"Evaluation Score": [], "Summary": [], "Score": []} #5
print("Starting evaluation...")
for eval_type, (criteria, steps) in evaluation_metrics.items():
for summ_type, summary in summaries.items(): #6
data["Evaluation Score"].append(eval_type) #6
data["Summary"].append(summ_type) #6
result = get_geval_score(criteria, steps, article_text, #6
summary, eval_type) #6
numeric_part = ''.join(filter(str.isdigit,
result.strip())) #7
if numeric_part:
score_num = int(float(result.strip()))
data["Score"].append(score_num) #8
max_values = {key: max(values) for key, values in data.items()}
df = pd.DataFrame(data) #9
pivot_df = df.pivot(index='Evaluation Score', #10
columns='Summary',
values='Score')
print(pivot_df)
Inspect the API workflow for assembling prompts and scoring each summary against each metric. section 12.3.2
Use G-Eval alongside conventional metrics and human review when judging suitability for real-world use. Its flexibility removes the need for reference summaries, but the result still depends on the criteria and prompts selected. section 12.3.1 section 12.6
