Evaluator
An evaluator is a model or component that assesses outputs against specified criteria and produces judgments, scores, comparisons, or feedback.
AI Agents in Action, Second Edition
LLM-as-judge is an evaluation technique in which a language model assesses another model's output against specified criteria.
- How it is used
- Checks agent outputs against criteria before release or feeds critique into another attempt.
- How deep it goes
- Develops rubric criteria, typed verdicts, critique loops, retry limits, and failure policies without descending into model internals.
- Examples
- Uses an image-style critic and exercises that score reasoning branches or retry after a test-tool failure.
- What you can do after
- Build it from scratch as a rubric-driven critic with typed feedback, bounded retries, and an explicit failure policy.
Read how AI Agents in Action, Second Edition explains evaluator →
Building Reliable AI Systems
An evaluation technique in which a large language model assesses the quality of another model's output.
- How it is used
- Assesses generated answers for hallucinations, relevance, and faithfulness in RAG evaluation and production quality checks.
- How deep it goes
- Provides a structured Python judging implementation, scoring criteria, and validation safeguards; does not establish automated verdicts as ground truth.
- Examples
- Uses a roaming-fee chatbot response and a Python reference-comparison function returning structured hallucination judgments.
- What you can do after
- Apply it via an OpenAI judge with Pydantic output, then validate disputed decisions with human review or fact-checking.
The Art of AI Product Development
LLM-as-a-Judge is an evaluation technique in which a large language model assesses generated outputs against specified criteria.
- How it is used
- Scales end-to-end RAG evaluation and checks whether fine-tuned models follow instructions.
- How deep it goes
- Defines groundedness and relevance criteria, then applies model scoring alongside manual audits and user feedback.
- Examples
- Tests a SaaS-trends report for grounding and relevance and scores instruction adherence on a held-out 20% test set.
- What you can do after
- Evaluate generated answers for source grounding and query relevance, or score how well a fine-tuned model follows instructions.
Read how The Art of AI Product Development explains evaluator →
Machine Learning Platform Engineering
LLM-as-judge is an evaluation technique in which one large language model assesses the outputs of another model.
- How it is used
- Evaluates model responses against application-specific quality criteria that conventional text-similarity metrics cannot assess.
- How deep it goes
- Defines G-Eval criteria, evaluation parameters, and score thresholds through production-oriented tests.
- Examples
- Tests security-boundary deflection and combines correctness, relevance, and completeness metrics.
- What you can do after
- Evaluate responses with DeepEval using custom G-Eval criteria and score thresholds.
Read how Machine Learning Platform Engineering explains evaluator →
Reinforcement Learning from Human Feedback
LLM-as-a-judge is an evaluation technique in which a language model assesses the quality or preference of model outputs according to provided criteria.
- How it is used
- Deploys language models to scale preference labeling, synthetic-data pipelines, and evaluation when human judgments are costly.
- How deep it goes
- Explains pairwise prompts, temperature control, and major biases, but stops short of a settled calibration recipe.
- Examples
- Uses judge prompts, MT-Bench, AlpacaEval, Arena-Hard, pairwise completions, and length-controlled evaluation variants.
- What you can do after
- Apply it via structured prompts to compare outputs, generate preference pairs, and scale evaluations while checking bias and uncertainty.
Read how Reinforcement Learning from Human Feedback explains evaluator →
Build an AI Agent (From Scratch)
LLM-as-a-judge is an evaluation method in which a language model scores or classifies another system's outputs according to an evaluation rubric.
- How it is used
- Scales agent-response evaluation across frequent prompt, model, and data changes by grading outputs against a rubric.
- How deep it goes
- Defines rubric-based scoring and pass/fail decisions, then explains single-output grading and pairwise comparison.
- Examples
- Uses a research-summary pass/fail rubric and a renewable-energy answer comparison.
- What you can do after
- Evaluate agent outputs with rubric-based scores or pass/fail decisions, or compare candidates and aggregate win rates.
Read how Build an AI Agent (From Scratch) explains evaluator →
Introduction to Generative AI, Second Edition
An LLM judge is a language model configured to assess or classify the outputs of another model according to specified criteria.
- How it is used
- Automates judgments about model outputs, especially for qualities that are subjective or difficult to verify deterministically.
- How deep it goes
- Explains score formats, expert-label validation, and precision and recall without detailing implementation.
- Examples
- Uses real estate listing positivity and illustrates binary, categorical, numeric, and ranking judgments.
- What you can do after
- Evaluate a judge by comparing its labels with expert judgments and checking precision and recall.
Read how Introduction to Generative AI, Second Edition explains evaluator →
AI Agents and Applications
- How it is used
- Tests LLM completions for relevance, correctness, and sensitivity as part of application evaluation.
- How deep it goes
- Names three evaluation criteria and dataset-based testing, but does not describe evaluator internals.
- Examples
- Uses relevance, correctness, and sensitivity as criteria for evaluating LLM completions.
- What you can do after
- Evaluate LLM completions with built-in checks for relevance, correctness, and sensitivity.
Building LLM Applications with DSPy
A language model assigned to assess the quality of responses produced by another language model.
- How it is used
- Uses model judgments as a metric during prediction evaluation and optimization.
- How deep it goes
- Shows a DSPy metric implementation and judge optimization, while leaving model-call cost controls to early stopping or minibatches.
- Examples
- Scores a predicted summary against a ground-truth summary and estimates calls for optimizing candidate prompts.
- What you can do after
- Apply it via a DSPy metric function that calls a judge to assess model predictions.
Read how Building LLM Applications with DSPy explains evaluator →
Related concepts
- language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- evaluation datasetuses · An evaluation dataset is a collection of examples or test cases, often with expected or judged outcomes, designated to assess a model or system against defined criteria.
- coherencemeasures · Coherence is the degree to which the parts of a text, reasoning process, or system fit together consistently and form a connected, intelligible whole.
- metric functionused by · A metric function is a function that evaluates a language-model prediction against an example and optionally a call trace to produce an assessment score.
- accuracymeasures · Accuracy is the degree to which an output, prediction, or measurement agrees with a correct or reference outcome under a specified criterion.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.