liveBook · Manning

G-Eval

As explained in Retrieval Augmented Generation, The Foundational Ideas

G-Eval is an LLM-based evaluation technique that generates evaluation steps and assigns probability-weighted scores to outputs.

G-Eval evaluates generated language outputs with a prompted LLM, generated evaluation steps, and probability-weighted scores. It helps distinguish output quality more finely than a raw 1–5 rating, but remains vulnerable to judge bias and run-to-run variation. section 14.3 section 14.3.1

A G-Eval evaluator combines three parts:

  • Defines an evaluation task and criteria, such as faithfulness or coherence on a 1–5 scale. section 14.3.1
  • Generates evaluation steps before assigning a score, prompting the model to identify claims, compare evidence, and assess quality systematically. section 14.3.1
  • Uses the model's probabilities for each possible rating rather than only its selected token, producing a continuous probability-weighted score. section 14.3.1

Faithfulness evaluation

A faithfulness prompt asks the evaluator to identify every factual claim in an answer, check each claim against the supplied context, count supported and unsupported claims, and assign a 1–5 score. The same pattern can evaluate RAG outputs without reference answers. section 14.3.1 section 14.7

score = ∑i=1npisi

The score is computed as the sum of each candidate rating multiplied by its probability, preserving uncertainty between adjacent ratings. section 14.3.1

G-Eval offers finer scoring with one LLM call plus chain-of-thought, but it does not provide confidence bounds. Check for self-preference, position or style bias, and stochastic inconsistency before treating differences as regressions. section 14.3.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains prompting, generated evaluation steps, probability-weighted scoring, and bias limitations, but does not provide confidence bounds.
Examples
Demonstrates coherence scoring, faithfulness checking, claim extraction, and probability-weighted ratings.
What it assumes
Assumes familiarity with LLM prompting, generated outputs, evaluation criteria, and source-context comparisons.
What you can do after
Apply G-Eval via an LLM client to produce fine-grained evaluation scores for generated answers and RAG outputs.
How it connects
Connects chain-of-thought, RAGAS, faithfulness, reference-free evaluation, and ARES-style trained evaluators.

Also explained in

Compare how 4 books explain G-Eval →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.