G-Eval uses a language model as a judge for generated text. It replaces a vague request for a quality score with a task description, explicit criteria, generated evaluation steps, and a constrained scoring form. The goal is a scalable proxy for human judgment, not a universal measure of quality. section 4.2 section 4.2.2
To evaluate a generated response:
- Define the task and one quality criterion. For example, ask for summary coherence on a 1–5 scale and describe what coherence means. section 4.2.2
- Generate evaluation steps from the task and criterion. For coherence, the steps can ask the judge to identify the source’s main points, compare the summary, and check their logical order. section 4.2.2 section 4.2.5
- Supply the source and generated text, then request a score in a constrained evaluation form. Repeat with separately defined criteria when assessing other dimensions. section 4.2.2 section 4.2.5
- When score-token probabilities are available, calculate a probability-weighted score across the allowed scores. This can distinguish outputs that receive the same integer score. section 4.2.3
Figure 4.4 The G-Eval evaluation pipeline. The framework transforms evaluation into a structured process: task introduction and criteria define the evaluation context, chain-of-thought steps are auto-generated, and probability-weighted scoring produces the final assessment.
The pipeline connects the criteria and generated steps to the final probability-weighted assessment. section 4.2.2
Evaluating a summary
A source reports a 15% revenue increase driven by cloud services and an 8% rise in operating costs. A candidate says revenue grew substantially because of cloud success, but higher expenses moderated the improvement. A coherence rubric can assess its organization; separate consistency and relevance rubrics can assess factual alignment and selection of important content. A single score should not stand in for all three. section 4.2.1 section 4.2.5
For probability weighting, multiply each allowed score by its probability and sum the results, normalizing over the available score probabilities. In one illustration, two summaries both receive a discrete 3, but their weighted scores are 3.21 and 2.92. If an API does not expose token probabilities, repeated independent evaluations at a nonzero temperature can be averaged as an approximation; this costs more evaluations. section 4.2.3
Reference-free evaluation does not mean ignoring available source material: a summary’s consistency still needs comparison with its source. Judge scores can also reflect verbosity or self-enhancement bias, and judges can be misled by errors. Compare judgments with human ratings and monitor them over time; use grounding material and human verification where appropriate. section 4.2.5 section 4.5
One reported academic-dataset result gives a Spearman correlation of 0.514 with human judgments for G-Eval, versus 0.17 for traditional metrics such as ROUGE. This is evidence for the evaluated setting, not a guarantee for every task; changing the task also requires changing its definition and criteria. section 4.2.5



