liveBook · Manning

LLM-as-a-judge

As explained in AI Agents in Action, Second Edition

LLM-as-judge is an evaluation technique in which a language model assesses another model's output against specified criteria.

LLM-as-a-judge uses a language model to assess another model’s output against specified criteria. Like a reviewer checking a draft, a separate judge can score or critique a candidate before it reaches the user. It is useful when several natural-language answers could be correct and quality is a matter of degree. section 1.3.5 section 7.1

A judge complements rather than replaces other checks. Use deterministic tests or numerical metrics when a schema, label, or known answer can be checked exactly. Use human review for cases requiring real-world judgment, and validate the rubric against human judgment so that consistent scores are not consistently wrong. section 7.1 section 7.3.4

A practical judging loop has four parts:

  • Define the rubric: identify the output’s purpose, choose specific and measurable criteria, and set a scoring scale and passing threshold. section 7.3.4 section 7.3.5
  • Give the judge the candidate and the context needed to assess it, such as the question, expected answer, or style guidelines. section 7.2.4 section 7.3.5
  • Return an inspectable result, such as a pass/fail value and feedback. Criterion-level critique explains what to improve while preserving what worked. section 7.2.4 section 7.3.5
  • Use the result to accept, revise, or block the output. If feedback triggers regeneration, cap retries and decide what happens when the limit is reached. section 7.3

Evaluating a retrieval answer

For each benchmark question, a retrieval agent produces a natural-language answer. A second agent receives the question, answer, expected key term, and a wrong-answer example. It returns a typed is_correct boolean and feedback. The key-term test can accept an answer that also contains unsupported extra claims, so add a separate grounding check when claims must be supported by retrieved context. section 7.2.4 section 7.3

Listing 7.5 04_RAG_agent.py (agents only)
agent = Agent(
    name="RAG Agent",
    instructions="""
You are a retrieval-augmented knowledge agent.
Break down the user's query into smaller parts if needed
to fetch relevant context for the user's query.    #1
""",
    tools=[search_knowledge_by_keyword],
    model="gpt-4o",  # Specify the model to use
)

class EvaluationOutput(BaseModel):    #2
    """Output model for evaluation agent."""
    is_correct: bool
    feedback: str

evaluation_agent = Agent(    #3
    name="Evaluation Agent",
    instructions="""
You are an evaluation agent.
Your task is to evaluate the answers provided by the RAG Agent.
You will compare the answer against 
the expected answers key term to validate correctness.
""",
    output_type=EvaluationOutput,
)

Inspect the answer agent and the evaluator’s typed pass/fail and feedback fields. section 7.2.4

Critiquing a generated image

An image critic assesses output against style guidelines and a five-point rubric. It returns a pass/fail result and feedback that can guide another generation attempt. Criteria such as color, lighting, and consistency make the critique more actionable than a bare pass or fail. section 7.3.5

A rubric only sees what its criteria ask about. Depending on the task, include grounding, completeness, tone, tool use, or error handling—not just surface quality. Rerun tests to check consistency, and track variability, latency, and token use. Maintain curated test sets and repeat judge checks after prompt or model changes. section 7.3.5 section 7.6 section 11.1.5

How AI Agents in Action, Second Edition covers it

How deep it goes
Develops rubric design and typed evaluation into bounded critic loops, with guidance on validation and failure handling.
Examples
Uses retrieval-answer benchmarks and an image-style critic to demonstrate typed feedback and rubric scoring.
What you can do after
Apply it via the Agents SDK to score outputs, return typed feedback, and guide bounded revisions.
How it connects
Connects rubric scoring to grounding, guardrails, benchmark testing, and human review.

Also explained in

Compare how 9 books explain LLM-as-a-judge →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Agents in Action, Second Edition. The section links point to the parts of the book it draws on.