G-Eval
G-Eval is an LLM-based technique for evaluating generated text against natural-language criteria, often by generating evaluation steps and assigning scores.
LLM Evaluation and Alignment, The Foundational Ideas
G-Eval is a framework for evaluating generated text by prompting a language model with task instructions, explicit criteria, generated evaluation steps, and a constrained scoring form.
- How it is used
- Uses an LLM judge to evaluate generated text against explicit task-specific criteria as a scalable proxy for human judgment.
- How deep it goes
- Explains structured prompting, generated evaluation steps, probability-weighted scoring, implementation considerations, and judge limitations.
- Examples
- Uses an earnings-summary coherence prompt and two equally rated summaries with different probability-weighted scores.
- What you can do after
- Apply it via structured prompts and probability-weighted scoring to assess generated text, then compare judge assessments with human judgments.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains G-Eval →
Machine Learning Platform Engineering
G-Eval is an evaluation technique that uses a large language model to judge another model’s outputs against natural-language criteria.
- How it is used
- Tests assistant responses against domain-specific quality and safety criteria.
- How deep it goes
- Works through criteria, test-case fields, thresholds, and combined metrics; does not detail the judge model’s internals.
- Examples
- Uses a security-sensitive credentials request, correctness and completeness tests, and documentation-grounded hallucination checks.
- What you can do after
- Apply it via DeepEval to test response quality and safety against configurable criteria and thresholds.
Read how Machine Learning Platform Engineering explains G-Eval →
Generative AI in Action
G-Eval is an LLM-based framework for evaluating generated text according to specified quality criteria without requiring reference texts.
- How it is used
- Assesses generated text against quality criteria when reference texts are unavailable.
- How deep it goes
- Explains prompt criteria, evaluation steps, scoring calls, and result comparison through a worked implementation.
- Examples
- Compares an NLP-generated summary and a Gemini-generated summary of an article about AI principles.
- What you can do after
- Apply it via Azure OpenAI GPT-4 to score and compare summaries across four criteria.
Retrieval Augmented Generation, The Foundational Ideas
G-Eval is an LLM-based evaluation technique that generates evaluation steps and assigns probability-weighted scores to outputs.
- How it is used
- Evaluates generated text and RAG outputs when reference answers are unavailable, using criteria such as faithfulness, relevance, or coherence.
- How deep it goes
- Explains prompting, generated evaluation steps, probability-weighted scoring, and bias limitations, but does not provide confidence bounds.
- Examples
- Demonstrates coherence scoring, faithfulness checking, claim extraction, and probability-weighted ratings.
- What you can do after
- Apply G-Eval via an LLM client to produce fine-grained evaluation scores for generated answers and RAG outputs.
Read how Retrieval Augmented Generation, The Foundational Ideas explains G-Eval →
Also covered in
- Context EngineeringIn early access (MEAP)
What to know first
- chain-of-thought promptingChain-of-thought prompting is a technique that elicits or supplies intermediate reasoning steps to guide a model toward a final answer.
- ROUGEROUGE is a family of automatic text-generation evaluation metrics that compare generated text with reference text using lexical overlap, often emphasizing recall.
- BLEU scoreBLEU is an automatic text-generation evaluation metric that compares candidate text with reference text using modified n-gram precision and a brevity penalty.
- fidelityFidelity is the degree to which an output, model, or representation accurately reflects or is supported by its relevant source, target, evidence, or behavior.
Related concepts
- chain-of-thought promptinguses · Chain-of-thought prompting is a technique that elicits or supplies intermediate reasoning steps to guide a model toward a final answer.
- GPT-4uses · GPT-4 is a multimodal large language model developed by OpenAI that processes text and image inputs and generates text.
- language modeluses · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- LLM-as-a-judgeimplements · LLM-as-a-judge is an evaluation technique in which a language model assesses or compares outputs according to specified criteria.
- probability theoryuses · Probability theory is the mathematical framework for quantifying and reasoning about uncertainty in events, variables, and outcomes.
- prompthas part · A prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
Questions about G-Eval
What is G-Eval?
G-Eval is an LLM-based technique for evaluating generated text against natural-language criteria, often by generating evaluation steps and assigning scores.
Which Manning books explain G-Eval?
LLM Evaluation and Alignment, The Foundational Ideas, Machine Learning Platform Engineering, Generative AI in Action and Retrieval Augmented Generation, The Foundational Ideas.
What should I know before learning G-Eval?
Manning books that teach G-Eval usually introduce chain-of-thought prompting, ROUGE, BLEU score and fidelity first.
What are the parts of G-Eval?
G-Eval includes prompt.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



