liveBook · Manning

G-Eval

G-Eval is an LLM-based technique for evaluating generated text against natural-language criteria, often by generating evaluation steps and assigning scores.

How 4 Manning books explain it

LLM Evaluation and Alignment, The Foundational Ideas

G-Eval is a framework for evaluating generated text by prompting a language model with task instructions, explicit criteria, generated evaluation steps, and a constrained scoring form.

How it is used
Uses an LLM judge to evaluate generated text against explicit task-specific criteria as a scalable proxy for human judgment.
How deep it goes
Explains structured prompting, generated evaluation steps, probability-weighted scoring, implementation considerations, and judge limitations.
Examples
Uses an earnings-summary coherence prompt and two equally rated summaries with different probability-weighted scores.
What you can do after
Apply it via structured prompts and probability-weighted scoring to assess generated text, then compare judge assessments with human judgments.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains G-Eval →

Machine Learning Platform Engineering

G-Eval is an evaluation technique that uses a large language model to judge another model’s outputs against natural-language criteria.

How it is used
Tests assistant responses against domain-specific quality and safety criteria.
How deep it goes
Works through criteria, test-case fields, thresholds, and combined metrics; does not detail the judge model’s internals.
Examples
Uses a security-sensitive credentials request, correctness and completeness tests, and documentation-grounded hallucination checks.
What you can do after
Apply it via DeepEval to test response quality and safety against configurable criteria and thresholds.

Read how Machine Learning Platform Engineering explains G-Eval →

Generative AI in Action

G-Eval is an LLM-based framework for evaluating generated text according to specified quality criteria without requiring reference texts.

How it is used
Assesses generated text against quality criteria when reference texts are unavailable.
How deep it goes
Explains prompt criteria, evaluation steps, scoring calls, and result comparison through a worked implementation.
Examples
Compares an NLP-generated summary and a Gemini-generated summary of an article about AI principles.
What you can do after
Apply it via Azure OpenAI GPT-4 to score and compare summaries across four criteria.

Read how Generative AI in Action explains G-Eval →

Retrieval Augmented Generation, The Foundational Ideas

G-Eval is an LLM-based evaluation technique that generates evaluation steps and assigns probability-weighted scores to outputs.

How it is used
Evaluates generated text and RAG outputs when reference answers are unavailable, using criteria such as faithfulness, relevance, or coherence.
How deep it goes
Explains prompting, generated evaluation steps, probability-weighted scoring, and bias limitations, but does not provide confidence bounds.
Examples
Demonstrates coherence scoring, faithfulness checking, claim extraction, and probability-weighted ratings.
What you can do after
Apply G-Eval via an LLM client to produce fine-grained evaluation scores for generated answers and RAG outputs.

Read how Retrieval Augmented Generation, The Foundational Ideas explains G-Eval →

Also covered in

What to know first

Related concepts

Questions about G-Eval

What is G-Eval?

G-Eval is an LLM-based technique for evaluating generated text against natural-language criteria, often by generating evaluation steps and assigning scores.

Which Manning books explain G-Eval?

LLM Evaluation and Alignment, The Foundational Ideas, Machine Learning Platform Engineering, Generative AI in Action and Retrieval Augmented Generation, The Foundational Ideas.

What should I know before learning G-Eval?

Manning books that teach G-Eval usually introduce chain-of-thought prompting, ROUGE, BLEU score and fidelity first.

What are the parts of G-Eval?

G-Eval includes prompt.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.