G-Eval
G-Eval is an LLM-based technique for evaluating generated text against natural-language criteria, often by generating evaluation steps and assigning scores.
Machine Learning Platform Engineering
G-Eval is an evaluation technique that uses a large language model to judge another model’s outputs against natural-language criteria.
- How it is used
- Tests assistant responses against domain-specific quality and safety criteria.
- How deep it goes
- Works through criteria, test-case fields, thresholds, and combined metrics; does not detail the judge model’s internals.
- Examples
- Uses a security-sensitive credentials request, correctness and completeness tests, and documentation-grounded hallucination checks.
- What you can do after
- Apply it via DeepEval to test response quality and safety against configurable criteria and thresholds.
Read how Machine Learning Platform Engineering explains G-Eval →
Generative AI in Action
G-Eval is an LLM-based framework for evaluating generated text according to specified quality criteria without requiring reference texts.
- How it is used
- Evaluates generated text when suitable reference answers are unavailable, including summaries and case-specific content.
- How deep it goes
- Works through criterion prompts, scoring steps, an API call, and score comparison without deriving model internals.
- Examples
- Compares an NLP-generated summary with an LLM-generated summary using relevance, coherence, consistency, and fluency scores.
- What you can do after
- Apply it via Azure OpenAI GPT-4 to score and compare summaries against defined criteria.
Related concepts
- chain-of-thought promptinguses · Chain-of-thought prompting is a technique that elicits or guides a model to produce or follow intermediate reasoning steps before a final answer.
- evaluatoruses · An evaluator is a model or component that assesses outputs against specified criteria and produces judgments, scores, comparisons, or feedback.
- token probabilityuses · Token probability is the probability a language model assigns to a candidate token at a particular position, conditioned on the preceding context.
- GPT-4uses · GPT-4 is a generative large language model developed by OpenAI for understanding and generating text and other supported inputs and outputs.
- language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- prompthas part · An input, typically containing instructions or context, supplied to an artificial-intelligence model to guide or elicit its output.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.