G-Eval evaluates generated language outputs with a prompted LLM, generated evaluation steps, and probability-weighted scores. It helps distinguish output quality more finely than a raw 1–5 rating, but remains vulnerable to judge bias and run-to-run variation. section 14.3 section 14.3.1
A G-Eval evaluator combines three parts:
- Defines an evaluation task and criteria, such as faithfulness or coherence on a 1–5 scale. section 14.3.1
- Generates evaluation steps before assigning a score, prompting the model to identify claims, compare evidence, and assess quality systematically. section 14.3.1
- Uses the model's probabilities for each possible rating rather than only its selected token, producing a continuous probability-weighted score. section 14.3.1
Faithfulness evaluation
A faithfulness prompt asks the evaluator to identify every factual claim in an answer, check each claim against the supplied context, count supported and unsupported claims, and assign a 1–5 score. The same pattern can evaluate RAG outputs without reference answers. section 14.3.1 section 14.7
The score is computed as the sum of each candidate rating multiplied by its probability, preserving uncertainty between adjacent ratings. section 14.3.1
G-Eval offers finer scoring with one LLM call plus chain-of-thought, but it does not provide confidence bounds. Check for self-preference, position or style bias, and stochastic inconsistency before treating differences as regressions. section 14.3.3



