liveBook · Manning

G-Eval

As explained in LLM Evaluation and Alignment, The Foundational Ideas

G-Eval is a framework for evaluating generated text by prompting a language model with task instructions, explicit criteria, generated evaluation steps, and a constrained scoring form.

G-Eval uses a language model as a judge for generated text. It replaces a vague request for a quality score with a task description, explicit criteria, generated evaluation steps, and a constrained scoring form. The goal is a scalable proxy for human judgment, not a universal measure of quality. section 4.2 section 4.2.2

To evaluate a generated response:

  • Define the task and one quality criterion. For example, ask for summary coherence on a 1–5 scale and describe what coherence means. section 4.2.2
  • Generate evaluation steps from the task and criterion. For coherence, the steps can ask the judge to identify the source’s main points, compare the summary, and check their logical order. section 4.2.2 section 4.2.5
  • Supply the source and generated text, then request a score in a constrained evaluation form. Repeat with separately defined criteria when assessing other dimensions. section 4.2.2 section 4.2.5
  • When score-token probabilities are available, calculate a probability-weighted score across the allowed scores. This can distinguish outputs that receive the same integer score. section 4.2.3
Figure 4.4 The G-Eval evaluation pipeline. The framework transforms evaluation into a structured process: task introduction and criteria define the evaluation context, chain-of-thought steps are auto-generated, and probability-weighted scoring produces the final assessment.

The pipeline connects the criteria and generated steps to the final probability-weighted assessment. section 4.2.2

Evaluating a summary

A source reports a 15% revenue increase driven by cloud services and an 8% rise in operating costs. A candidate says revenue grew substantially because of cloud success, but higher expenses moderated the improvement. A coherence rubric can assess its organization; separate consistency and relevance rubrics can assess factual alignment and selection of important content. A single score should not stand in for all three. section 4.2.1 section 4.2.5

For probability weighting, multiply each allowed score by its probability and sum the results, normalizing over the available score probabilities. In one illustration, two summaries both receive a discrete 3, but their weighted scores are 3.21 and 2.92. If an API does not expose token probabilities, repeated independent evaluations at a nonzero temperature can be averaged as an approximation; this costs more evaluations. section 4.2.3

Reference-free evaluation does not mean ignoring available source material: a summary’s consistency still needs comparison with its source. Judge scores can also reflect verbosity or self-enhancement bias, and judges can be misled by errors. Compare judgments with human ratings and monitor them over time; use grounding material and human verification where appropriate. section 4.2.5 section 4.5

One reported academic-dataset result gives a Spearman correlation of 0.514 with human judgments for G-Eval, versus 0.17 for traditional metrics such as ROUGE. This is evidence for the evaluated setting, not a guarantee for every task; changing the task also requires changing its definition and criteria. section 4.2.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains structured prompting, generated evaluation steps, probability-weighted scoring, implementation considerations, and judge limitations.
Examples
Uses an earnings-summary coherence prompt and two equally rated summaries with different probability-weighted scores.
What you can do after
Apply it via structured prompts and probability-weighted scoring to assess generated text, then compare judge assessments with human judgments.
How it connects
Connects LLM-as-a-judge with chain-of-thought prompting, human evaluation, ROUGE, and Spearman correlation.

Also explained in

Compare how 4 books explain G-Eval →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.