An LLM-as-judge evaluator is a separate model that assesses an agent’s output against specified criteria. It can score or critique a candidate before the agent returns it, or provide feedback that guides another attempt. Think of it as a reviewer with a grading rubric, not as proof that the answer is correct. section 1.3.5 section 7.3.4
Choose the check to match the output:
- For structured outputs with known targets, use metrics such as accuracy, precision, recall, or F1 rather than adding a judge unnecessarily. section 7.3.4
- For open-ended summaries, explanations, or plans, use a rubric and an evaluator when several phrasings could be valid. section 7.3.4
- For answers that must match retrieved context, use a grounding agent to check claims against that source. A critic instead checks generated content against criteria; an evaluation agent handles more complex output. section 7.1 section 7.3
A useful rubric starts with the output’s purpose, then names specific criteria and a scoring scale. Ask for feedback as well as a pass/fail result: a bare verdict says little about what to fix. Criteria might cover grounding, completeness, tone, tool use, or error handling. Check the rubric against human judgment; a judge can apply an unsuitable rubric consistently. section 7.3.4 section 7.3.5
Reviewing a generated image
An image agent receives style guidelines for color, lighting, and other features. A critic receives those guidelines and a five-point rubric, then returns whether the image passes and what should change. The example sets the passing threshold at 2 or higher; that threshold is a design choice, not a universal quality standard. section 7.3.5
Listing 7.12 08_image_vision_critic_agents.py (critic agent)
class CritqueImage(BaseModel): #1
"""Result of critiquing image."""
image_pass: bool
feedback: str
critic = Agent(
name="Image Critic",
instructions=f"""You are an image critic.
Your task is to evaluate the quality of generated images
based on the provided specific criteria and style guidelines.
{style_guidelines} #2
{rubric} #3
""",
model="gpt-5-mini",
tools=[describe_image], #4
output_type=CritqueImage, #5
)
# critic loop
while failing := True: #6
input = dict(
description=image_description,
feedback=feedback, #7
)
result = await Runner.run(agent, str(input))
# code to save the image ommited
critique_result = await Runner.run(
critic,
f"Please critique the image at {image_path} with the prompt: {image_description}",
)
critique = critique_result.final_output
failing = critique.image_pass is False #6
feedback = critique.feedback #7
The critic-agent listing shows guidelines and a rubric supplied to the judge, with typed pass/fail and feedback fields that a program can inspect. section 7.3.5
Make the verdict operational: pass acceptable output, or use the critique to regenerate and check again. Set an attempt limit before starting. When retries run out, choose an explicit fallback, error, confidence flag, or human escalation according to the stakes; do not silently return a failing result. Different evaluator models can reduce shared blind spots, though human review remains important. section 7.3 section 7.1
For deployed agents, run judge checks on curated test sets after prompt or model changes. Trace outcomes, latency, and token use, and rerun benchmarks to see whether scores are stable. Use the findings to revise prompts, tools, policies, or the rubric itself. section 11.1.5 section 7.6
