LLM-as-a-judge
LLM-as-a-judge is an evaluation technique in which a language model assesses or compares outputs according to specified criteria.
LLM Evaluation and Alignment, The Foundational Ideas
An AI labeler is a language model that evaluates or compares outputs and generates labels for alignment data.
- How it is used
- Generates scalable quality judgments and preference labels for evaluating open-ended outputs and producing alignment data.
- How deep it goes
- Explains rubric design, pairwise preference generation, human calibration, deployment decisions, and bias auditing.
- Examples
- Uses customer-service response assessment and principle-guided A/B comparisons to illustrate evaluator judgments.
- What you can do after
- Apply an LLM judge to compare responses using a rubric, calibrate its labels against human judgments, and identify deployment risks.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains LLM-as-a-judge →
AI Model Evaluation
An evaluation method in which a large language model assesses another model's outputs or behaviors and assigns judgments or scores.
- How it is used
- Provides scalable qualitative scoring and diagnostics for open-ended outputs when defined quality criteria exceed simpler metrics.
- How deep it goes
- Develops a procedural evaluation workflow from goal and rubric design through calibration, validation, model selection, and operational monitoring.
- Examples
- Uses a movie recommender’s top-five outputs and viewing history to illustrate relevance, diversity, and an underspecified judging prompt.
- What you can do after
- Evaluate and understand judge-assisted assessments by designing rubrics, choosing comparison formats, and validating judgments against known examples or human ratings.
AI Agents in Action, Second Edition
LLM-as-judge is an evaluation technique in which a language model assesses another model's output against specified criteria.
- How it is used
- Evaluates agent outputs with rubric-based scores or critiques before delivery or as feedback for further action.
- How deep it goes
- Develops rubric design and typed evaluation into bounded critic loops, with guidance on validation and failure handling.
- Examples
- Uses retrieval-answer benchmarks and an image-style critic to demonstrate typed feedback and rubric scoring.
- What you can do after
- Apply it via the Agents SDK to score outputs, return typed feedback, and guide bounded revisions.
Read how AI Agents in Action, Second Edition explains LLM-as-a-judge →
Vibe Engineering
LLM-as-a-Judge is an evaluation approach in which a large language model assesses or grades an artifact, change, or another model's output against specified criteria.
- How it is used
- Evaluates generated work and code changes against criteria to widen review coverage without replacing human review.
- How deep it goes
- Explains judge limitations and review workflows, including advisory hooks, scoped agents, and external verification signals.
- Examples
- Illustrates advisory pre-commit review, dependency validation, and error detection in generated COBOL.
- What you can do after
- Operate advisory Git-hook or CI reviewers with explicit criteria while retaining deterministic checks and human review.
Retrieval Augmented Generation, The Foundational Ideas
An LLM judge is a large language model configured to evaluate the quality or correctness of generated outputs.
- How it is used
- Assesses RAG answers or retrieved context, and can supply training labels or reject inadequate context before generation.
- How deep it goes
- Distinguishes prompted, trained, and pipeline critics, then examines selection tradeoffs and documented biases without specifying a universal grading rubric.
- Examples
- Uses a mortgage-compliance faithfulness regression, Self-RAG’s critic-labeled training data, and a graph-retrieval retry path.
- What you can do after
- Evaluate/understand when a judge can diagnose RAG failures or control a pipeline, and check its bias before trusting scores.
Read how Retrieval Augmented Generation, The Foundational Ideas explains LLM-as-a-judge →
Evaluating AI Systems
An AI system that assesses outputs or behavior according to evaluation criteria.
- How it is used
- Uses AI judgments for scalable model comparison, stress testing, and production monitoring, with human review for high-stakes decisions.
- How deep it goes
- Gives prompt-writing and model-selection guidance, with cautions on validation, bias, and metric drift.
- Examples
- Illustrates a coherence judge that compares a news summary with its source and assigns a 1–5 score.
- What you can do after
- Build an LLM judge from a rubric, select a model, and constrain its judgment instructions and output.
Self-Improving Agents
A pattern in which a large language model evaluates outputs against a rubric, potentially comparing candidates to select the best.
- How it is used
- Scores agent outputs against a rubric and compares prompt variants within an offline improvement loop.
- How deep it goes
- Works through pairwise inputs, ordered rubric criteria, swap-and-agree, and disagreement handling; stops short of treating judge verdicts as ground truth.
- Examples
- Uses a late-return support answer and a grounded-versus-hallucinated research answer, plus exercises that swap positions and reorder rubric criteria.
- What you can do after
- Build it from scratch: implement a pairwise judge with swap-and-agree, then use its verdict to select offline prompt variants.
LLM Customization and Fine-Tuning
An evaluation setup in which a language model judges other model outputs against specified criteria.
- How it is used
- Evaluates open-ended outputs and generated data against criteria that token overlap or exact matching can miss.
- How deep it goes
- Specifies judge inputs, scoring dimensions, test-set comparison, model selection, and evaluation cadence without providing a judge implementation.
- Examples
- Compares VPN instruction paraphrases and invented steps, then scores student responses for correctness, completeness, and coherence.
- What you can do after
- Evaluate model outputs using a rubric, compare judge scores with Token-F1 by category, and identify possible surface matching or successful paraphrasing.
Read how LLM Customization and Fine-Tuning explains LLM-as-a-judge →
Context Engineering
An evaluator is a component that assesses outcomes against specified criteria or metrics.
- How it is used
- Assesses candidate outputs against natural-language rubrics when semantic qualities are difficult to score with simple heuristics.
- How deep it goes
- Explains judge inputs, structured outputs, scoring patterns, biases, and a library-based evaluation workflow.
- Examples
- Evaluates a small RAG benchmark using faithfulness, answer relevancy, context precision, and context recall.
- What you can do after
- Apply it via Ragas to generate evaluation signals and calibrate them before using them as release gates.
Building Reliable AI Systems
An evaluation technique in which a large language model assesses the quality of another model's output.
- How it is used
- Uses model judgments to assess quality, relevance, faithfulness, and hallucinations in generated outputs.
- How deep it goes
- Explains evaluation inputs, criteria, structured judgments, biases, and practical checks, with a Python implementation.
- Examples
- Uses roaming-fee responses and a structured hallucination judgment with severity, explanation, and unsupported claims.
- What you can do after
- Apply it via the OpenAI API and Pydantic to compare generated text with a reference and return a structured judgment.
Read how Building Reliable AI Systems explains LLM-as-a-judge →
The Art of AI Product Development
LLM-as-a-Judge is an evaluation technique in which a large language model assesses generated outputs against specified criteria.
- How it is used
- Evaluates end-to-end RAG responses and instruction-following outputs against quality criteria.
- How deep it goes
- Defines groundedness and relevance, gives a held-out-set example, and notes oversight without detailing evaluator implementation.
- Examples
- Evaluates an AI-trends report for grounding and relevance, and scores instruction adherence on a held-out test set.
- What you can do after
- Evaluate generated responses for grounding, relevance, or instruction adherence using a language-model judge.
Read how The Art of AI Product Development explains LLM-as-a-judge →
Machine Learning Platform Engineering
LLM-as-judge is an evaluation technique in which one large language model assesses the outputs of another model.
- How it is used
- Evaluates model responses against application-specific quality criteria that conventional text-similarity metrics cannot assess.
- How deep it goes
- Defines G-Eval criteria, evaluation parameters, and score thresholds through production-oriented tests.
- Examples
- Tests security-boundary deflection and combines correctness, relevance, and completeness metrics.
- What you can do after
- Evaluate responses with DeepEval using custom G-Eval criteria and score thresholds.
Read how Machine Learning Platform Engineering explains LLM-as-a-judge →
AI Governance
A language model used to evaluate another model’s plan or output against specified criteria.
- How it is used
- Uses a second model to flag hallucinations or agent-plan deviations for regeneration, human review, or escalation.
- How deep it goes
- Explains judge validation, prompt tuning, cost, and bias, but does not specify a judge architecture.
- Examples
- Uses ClaimAssist judge comparisons against human-labeled data and checks agent plans against approved versions.
- What you can do after
- Evaluate judges against human decisions, tune prompts, and route flagged outputs or plans to review.
Build an Advanced RAG Application (From Scratch)
LLM-as-judge evaluation uses a language model to assess the quality, factual consistency, or other properties of generated responses.
- How it is used
- Deploys language models to evaluate RAG response quality, faithfulness, factual consistency, and hallucination.
- How deep it goes
- Contrasts model-based judgment with classification and rule-based evaluation but does not specify prompts or implementation steps.
- Examples
- Uses RAGAS faithfulness, HHEM comparison, Vertex AI evaluation, and production-aligned metric testing as concrete evaluation contexts.
- What you can do after
- Evaluate RAG responses with nuanced model-based judgments and compare system changes against selected metrics.
Read how Build an Advanced RAG Application (From Scratch) explains LLM-as-a-judge →
Reinforcement Learning from Human Feedback
LLM-as-a-judge is an evaluation technique in which a language model assesses the quality or preference of model outputs according to provided criteria.
- How it is used
- Uses prompted language models to label preferences and scale response evaluation more cheaply than human judgment.
- How deep it goes
- Defines pairwise prompting and variance control, then discusses bias and alternatives without prescribing a full calibration procedure.
- Examples
- Uses an MT-Bench pairwise prompt and names MT-Bench, AlpacaEval, Arena-Hard, and WildBench.
- What you can do after
- Apply prompting with judging instructions, a user prompt, and two completions to collect a preference or evaluation.
Read how Reinforcement Learning from Human Feedback explains LLM-as-a-judge →
Build an AI Agent (From Scratch)
LLM-as-a-judge is an evaluation method in which a language model scores or classifies another system's outputs according to an evaluation rubric.
- How it is used
- Scales evaluation of agent responses by grading them against a rubric as prompts, models, and data change.
- How deep it goes
- Distinguishes single-output grading from pairwise comparison and gives prompt examples, without specifying judge-model implementation.
- Examples
- Uses a research-summary PASS/FAIL prompt and a renewable-energy A/B/Tie comparison.
- What you can do after
- Evaluate agent responses with rubric-based scores or pairwise choices, then aggregate comparisons into win rates.
Read how Build an AI Agent (From Scratch) explains LLM-as-a-judge →
Introduction to Generative AI, Second Edition
An LLM judge is a language model configured to assess or classify the outputs of another model according to specified criteria.
- How it is used
- Automates human judgments of model outputs to support systematic evaluation of prompt iterations.
- How deep it goes
- Works through expert-label validation and precision and recall, while noting that accuracy can mislead with imbalanced labels.
- Examples
- Uses real estate listing descriptions and judges whether their tone is appropriately enthusiastic and positive.
- What you can do after
- Evaluate whether a judge’s labels resemble expert labels by checking agreement, precision, and recall.
Read how Introduction to Generative AI, Second Edition explains LLM-as-a-judge →
Designing AI Systems
An LLM judge is a language model used to evaluate another model's output against specified criteria.
- How it is used
- Adds model-based quality scoring for subjective output qualities within an observability or evaluation pipeline.
- How deep it goes
- Defines scorer and score structures and explains tradeoffs, sampling, and calibration without detailing judge prompts.
- Examples
- Contrasts subjective qualities with structural checks and illustrates potential self-style bias using GPT-4o.
- What you can do after
- Apply it through a scorer interface to attach model-judged quality scores to traces, spans, or generations.
AI Agents and Applications
- How it is used
- Supports testing LLM application outputs against evaluation criteria.
- How deep it goes
- Names evaluation criteria but does not detail evaluator implementation.
- Examples
- Uses relevance, correctness, and sensitivity as criteria for evaluating LLM completions.
- What you can do after
- Apply via LangSmith built-in evaluators to test completions for relevance, correctness, and sensitivity.
Read how AI Agents and Applications explains LLM-as-a-judge →
Building LLM Applications with DSPy
A language model assigned to assess the quality of responses produced by another language model.
- How it is used
- Scores model predictions inside metric functions during optimization and evaluation, rather than production inference.
- How deep it goes
- Implements a DSPy Predict judge that returns a quality score and can be prompt-optimized with demos and instructions.
- Examples
- Compares generated summaries with ground-truth summaries and illustrates judge-call costs across candidate prompts.
- What you can do after
- Evaluate summarizer predictions with a judge-based metric and optimize the judge’s demos and instructions.
Read how Building LLM Applications with DSPy explains LLM-as-a-judge →
Designing AI Agents
Model-based judging is the assessment of outputs using a model as the judge.
- How it is used
- Scores model outputs as an explicit quality measurement alongside operational metrics.
- How deep it goes
- Describes separate-model scoring and calibration needs without specifying a scoring procedure.
- What you can do after
- Evaluate outputs with a separate model while treating its score as one measurement rather than ground truth.
Also covered in
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- human-in-the-loopHuman-in-the-loop is a system or workflow design in which people participate in reviewing, guiding, correcting, or authorizing an automated process.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- promptA prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
Related concepts
- language modeluses · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- evaluation datasetuses · An evaluation dataset is a collection of data or test cases used to assess a model or system against specified outcomes or criteria.
- large language modeluses · A large language model is a machine-learning model trained on extensive text data to process and generate natural language.
- accuracymeasures · Accuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- coherencemeasures · Coherence is the degree to which the parts of an output are logically connected, consistent, and intelligible as a whole.
- metric functionused by · A metric function is a function that evaluates a language-model prediction against an example and optionally a call trace to produce an assessment score.
Questions about LLM-as-a-judge
What is LLM-as-a-judge?
LLM-as-a-judge is an evaluation technique in which a language model assesses or compares outputs according to specified criteria.
Which Manning books explain LLM-as-a-judge?
LLM Evaluation and Alignment, The Foundational Ideas, AI Model Evaluation, AI Agents in Action, Second Edition, Vibe Engineering, Retrieval Augmented Generation, The Foundational Ideas, Evaluating AI Systems, Self-Improving Agents, LLM Customization and Fine-Tuning and 13 more.
What should I know before learning LLM-as-a-judge?
Manning books that teach LLM-as-a-judge usually introduce language model, human-in-the-loop, accuracy and prompt first.
What is LLM-as-a-judge also called?
LLM-as-a-judge is also called LLM-as-judge.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




















