liveBook · Manning

LLM-as-a-judge

Also called LLM-as-judge

LLM-as-a-judge is an evaluation technique in which a language model assesses or compares outputs according to specified criteria.

How 21 Manning books explain it

LLM Evaluation and Alignment, The Foundational Ideas

An AI labeler is a language model that evaluates or compares outputs and generates labels for alignment data.

How it is used
Generates scalable quality judgments and preference labels for evaluating open-ended outputs and producing alignment data.
How deep it goes
Explains rubric design, pairwise preference generation, human calibration, deployment decisions, and bias auditing.
Examples
Uses customer-service response assessment and principle-guided A/B comparisons to illustrate evaluator judgments.
What you can do after
Apply an LLM judge to compare responses using a rubric, calibrate its labels against human judgments, and identify deployment risks.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains LLM-as-a-judge →

AI Model Evaluation

An evaluation method in which a large language model assesses another model's outputs or behaviors and assigns judgments or scores.

How it is used
Provides scalable qualitative scoring and diagnostics for open-ended outputs when defined quality criteria exceed simpler metrics.
How deep it goes
Develops a procedural evaluation workflow from goal and rubric design through calibration, validation, model selection, and operational monitoring.
Examples
Uses a movie recommender’s top-five outputs and viewing history to illustrate relevance, diversity, and an underspecified judging prompt.
What you can do after
Evaluate and understand judge-assisted assessments by designing rubrics, choosing comparison formats, and validating judgments against known examples or human ratings.

Read how AI Model Evaluation explains LLM-as-a-judge →

AI Agents in Action, Second Edition

LLM-as-judge is an evaluation technique in which a language model assesses another model's output against specified criteria.

How it is used
Evaluates agent outputs with rubric-based scores or critiques before delivery or as feedback for further action.
How deep it goes
Develops rubric design and typed evaluation into bounded critic loops, with guidance on validation and failure handling.
Examples
Uses retrieval-answer benchmarks and an image-style critic to demonstrate typed feedback and rubric scoring.
What you can do after
Apply it via the Agents SDK to score outputs, return typed feedback, and guide bounded revisions.

Read how AI Agents in Action, Second Edition explains LLM-as-a-judge →

Vibe Engineering

LLM-as-a-Judge is an evaluation approach in which a large language model assesses or grades an artifact, change, or another model's output against specified criteria.

How it is used
Evaluates generated work and code changes against criteria to widen review coverage without replacing human review.
How deep it goes
Explains judge limitations and review workflows, including advisory hooks, scoped agents, and external verification signals.
Examples
Illustrates advisory pre-commit review, dependency validation, and error detection in generated COBOL.
What you can do after
Operate advisory Git-hook or CI reviewers with explicit criteria while retaining deterministic checks and human review.

Read how Vibe Engineering explains LLM-as-a-judge →

Retrieval Augmented Generation, The Foundational Ideas

An LLM judge is a large language model configured to evaluate the quality or correctness of generated outputs.

How it is used
Assesses RAG answers or retrieved context, and can supply training labels or reject inadequate context before generation.
How deep it goes
Distinguishes prompted, trained, and pipeline critics, then examines selection tradeoffs and documented biases without specifying a universal grading rubric.
Examples
Uses a mortgage-compliance faithfulness regression, Self-RAG’s critic-labeled training data, and a graph-retrieval retry path.
What you can do after
Evaluate/understand when a judge can diagnose RAG failures or control a pipeline, and check its bias before trusting scores.

Read how Retrieval Augmented Generation, The Foundational Ideas explains LLM-as-a-judge →

Evaluating AI Systems

An AI system that assesses outputs or behavior according to evaluation criteria.

How it is used
Uses AI judgments for scalable model comparison, stress testing, and production monitoring, with human review for high-stakes decisions.
How deep it goes
Gives prompt-writing and model-selection guidance, with cautions on validation, bias, and metric drift.
Examples
Illustrates a coherence judge that compares a news summary with its source and assigns a 1–5 score.
What you can do after
Build an LLM judge from a rubric, select a model, and constrain its judgment instructions and output.

Read how Evaluating AI Systems explains LLM-as-a-judge →

Self-Improving Agents

A pattern in which a large language model evaluates outputs against a rubric, potentially comparing candidates to select the best.

How it is used
Scores agent outputs against a rubric and compares prompt variants within an offline improvement loop.
How deep it goes
Works through pairwise inputs, ordered rubric criteria, swap-and-agree, and disagreement handling; stops short of treating judge verdicts as ground truth.
Examples
Uses a late-return support answer and a grounded-versus-hallucinated research answer, plus exercises that swap positions and reorder rubric criteria.
What you can do after
Build it from scratch: implement a pairwise judge with swap-and-agree, then use its verdict to select offline prompt variants.

Read how Self-Improving Agents explains LLM-as-a-judge →

LLM Customization and Fine-Tuning

An evaluation setup in which a language model judges other model outputs against specified criteria.

How it is used
Evaluates open-ended outputs and generated data against criteria that token overlap or exact matching can miss.
How deep it goes
Specifies judge inputs, scoring dimensions, test-set comparison, model selection, and evaluation cadence without providing a judge implementation.
Examples
Compares VPN instruction paraphrases and invented steps, then scores student responses for correctness, completeness, and coherence.
What you can do after
Evaluate model outputs using a rubric, compare judge scores with Token-F1 by category, and identify possible surface matching or successful paraphrasing.

Read how LLM Customization and Fine-Tuning explains LLM-as-a-judge →

Context Engineering

An evaluator is a component that assesses outcomes against specified criteria or metrics.

How it is used
Assesses candidate outputs against natural-language rubrics when semantic qualities are difficult to score with simple heuristics.
How deep it goes
Explains judge inputs, structured outputs, scoring patterns, biases, and a library-based evaluation workflow.
Examples
Evaluates a small RAG benchmark using faithfulness, answer relevancy, context precision, and context recall.
What you can do after
Apply it via Ragas to generate evaluation signals and calibrate them before using them as release gates.

Read how Context Engineering explains LLM-as-a-judge →

Building Reliable AI Systems

An evaluation technique in which a large language model assesses the quality of another model's output.

How it is used
Uses model judgments to assess quality, relevance, faithfulness, and hallucinations in generated outputs.
How deep it goes
Explains evaluation inputs, criteria, structured judgments, biases, and practical checks, with a Python implementation.
Examples
Uses roaming-fee responses and a structured hallucination judgment with severity, explanation, and unsupported claims.
What you can do after
Apply it via the OpenAI API and Pydantic to compare generated text with a reference and return a structured judgment.

Read how Building Reliable AI Systems explains LLM-as-a-judge →

The Art of AI Product Development

LLM-as-a-Judge is an evaluation technique in which a large language model assesses generated outputs against specified criteria.

How it is used
Evaluates end-to-end RAG responses and instruction-following outputs against quality criteria.
How deep it goes
Defines groundedness and relevance, gives a held-out-set example, and notes oversight without detailing evaluator implementation.
Examples
Evaluates an AI-trends report for grounding and relevance, and scores instruction adherence on a held-out test set.
What you can do after
Evaluate generated responses for grounding, relevance, or instruction adherence using a language-model judge.

Read how The Art of AI Product Development explains LLM-as-a-judge →

Machine Learning Platform Engineering

LLM-as-judge is an evaluation technique in which one large language model assesses the outputs of another model.

How it is used
Evaluates model responses against application-specific quality criteria that conventional text-similarity metrics cannot assess.
How deep it goes
Defines G-Eval criteria, evaluation parameters, and score thresholds through production-oriented tests.
Examples
Tests security-boundary deflection and combines correctness, relevance, and completeness metrics.
What you can do after
Evaluate responses with DeepEval using custom G-Eval criteria and score thresholds.

Read how Machine Learning Platform Engineering explains LLM-as-a-judge →

AI Governance

A language model used to evaluate another model’s plan or output against specified criteria.

How it is used
Uses a second model to flag hallucinations or agent-plan deviations for regeneration, human review, or escalation.
How deep it goes
Explains judge validation, prompt tuning, cost, and bias, but does not specify a judge architecture.
Examples
Uses ClaimAssist judge comparisons against human-labeled data and checks agent plans against approved versions.
What you can do after
Evaluate judges against human decisions, tune prompts, and route flagged outputs or plans to review.

Read how AI Governance explains LLM-as-a-judge →

Build an Advanced RAG Application (From Scratch)

LLM-as-judge evaluation uses a language model to assess the quality, factual consistency, or other properties of generated responses.

How it is used
Deploys language models to evaluate RAG response quality, faithfulness, factual consistency, and hallucination.
How deep it goes
Contrasts model-based judgment with classification and rule-based evaluation but does not specify prompts or implementation steps.
Examples
Uses RAGAS faithfulness, HHEM comparison, Vertex AI evaluation, and production-aligned metric testing as concrete evaluation contexts.
What you can do after
Evaluate RAG responses with nuanced model-based judgments and compare system changes against selected metrics.

Read how Build an Advanced RAG Application (From Scratch) explains LLM-as-a-judge →

Reinforcement Learning from Human Feedback

LLM-as-a-judge is an evaluation technique in which a language model assesses the quality or preference of model outputs according to provided criteria.

How it is used
Uses prompted language models to label preferences and scale response evaluation more cheaply than human judgment.
How deep it goes
Defines pairwise prompting and variance control, then discusses bias and alternatives without prescribing a full calibration procedure.
Examples
Uses an MT-Bench pairwise prompt and names MT-Bench, AlpacaEval, Arena-Hard, and WildBench.
What you can do after
Apply prompting with judging instructions, a user prompt, and two completions to collect a preference or evaluation.

Read how Reinforcement Learning from Human Feedback explains LLM-as-a-judge →

Build an AI Agent (From Scratch)

LLM-as-a-judge is an evaluation method in which a language model scores or classifies another system's outputs according to an evaluation rubric.

How it is used
Scales evaluation of agent responses by grading them against a rubric as prompts, models, and data change.
How deep it goes
Distinguishes single-output grading from pairwise comparison and gives prompt examples, without specifying judge-model implementation.
Examples
Uses a research-summary PASS/FAIL prompt and a renewable-energy A/B/Tie comparison.
What you can do after
Evaluate agent responses with rubric-based scores or pairwise choices, then aggregate comparisons into win rates.

Read how Build an AI Agent (From Scratch) explains LLM-as-a-judge →

Introduction to Generative AI, Second Edition

An LLM judge is a language model configured to assess or classify the outputs of another model according to specified criteria.

How it is used
Automates human judgments of model outputs to support systematic evaluation of prompt iterations.
How deep it goes
Works through expert-label validation and precision and recall, while noting that accuracy can mislead with imbalanced labels.
Examples
Uses real estate listing descriptions and judges whether their tone is appropriately enthusiastic and positive.
What you can do after
Evaluate whether a judge’s labels resemble expert labels by checking agreement, precision, and recall.

Read how Introduction to Generative AI, Second Edition explains LLM-as-a-judge →

Designing AI Systems

An LLM judge is a language model used to evaluate another model's output against specified criteria.

How it is used
Adds model-based quality scoring for subjective output qualities within an observability or evaluation pipeline.
How deep it goes
Defines scorer and score structures and explains tradeoffs, sampling, and calibration without detailing judge prompts.
Examples
Contrasts subjective qualities with structural checks and illustrates potential self-style bias using GPT-4o.
What you can do after
Apply it through a scorer interface to attach model-judged quality scores to traces, spans, or generations.

Read how Designing AI Systems explains LLM-as-a-judge →

AI Agents and Applications

How it is used
Supports testing LLM application outputs against evaluation criteria.
How deep it goes
Names evaluation criteria but does not detail evaluator implementation.
Examples
Uses relevance, correctness, and sensitivity as criteria for evaluating LLM completions.
What you can do after
Apply via LangSmith built-in evaluators to test completions for relevance, correctness, and sensitivity.

Read how AI Agents and Applications explains LLM-as-a-judge →

Building LLM Applications with DSPy

A language model assigned to assess the quality of responses produced by another language model.

How it is used
Scores model predictions inside metric functions during optimization and evaluation, rather than production inference.
How deep it goes
Implements a DSPy Predict judge that returns a quality score and can be prompt-optimized with demos and instructions.
Examples
Compares generated summaries with ground-truth summaries and illustrates judge-call costs across candidate prompts.
What you can do after
Evaluate summarizer predictions with a judge-based metric and optimize the judge’s demos and instructions.

Read how Building LLM Applications with DSPy explains LLM-as-a-judge →

Designing AI Agents

Model-based judging is the assessment of outputs using a model as the judge.

How it is used
Scores model outputs as an explicit quality measurement alongside operational metrics.
How deep it goes
Describes separate-model scoring and calibration needs without specifying a scoring procedure.
What you can do after
Evaluate outputs with a separate model while treating its score as one measurement rather than ground truth.

Read how Designing AI Agents explains LLM-as-a-judge →

Also covered in

What to know first

Related concepts

Questions about LLM-as-a-judge

What is LLM-as-a-judge?

LLM-as-a-judge is an evaluation technique in which a language model assesses or compares outputs according to specified criteria.

Which Manning books explain LLM-as-a-judge?

LLM Evaluation and Alignment, The Foundational Ideas, AI Model Evaluation, AI Agents in Action, Second Edition, Vibe Engineering, Retrieval Augmented Generation, The Foundational Ideas, Evaluating AI Systems, Self-Improving Agents, LLM Customization and Fine-Tuning and 13 more.

What should I know before learning LLM-as-a-judge?

Manning books that teach LLM-as-a-judge usually introduce language model, human-in-the-loop, accuracy and prompt first.

What is LLM-as-a-judge also called?

LLM-as-a-judge is also called LLM-as-judge.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.