LLM-as-a-judge uses a language model to assess another model’s output against specified criteria. Like a reviewer checking a draft, a separate judge can score or critique a candidate before it reaches the user. It is useful when several natural-language answers could be correct and quality is a matter of degree. section 1.3.5 section 7.1
A judge complements rather than replaces other checks. Use deterministic tests or numerical metrics when a schema, label, or known answer can be checked exactly. Use human review for cases requiring real-world judgment, and validate the rubric against human judgment so that consistent scores are not consistently wrong. section 7.1 section 7.3.4
A practical judging loop has four parts:
- Define the rubric: identify the output’s purpose, choose specific and measurable criteria, and set a scoring scale and passing threshold. section 7.3.4 section 7.3.5
- Give the judge the candidate and the context needed to assess it, such as the question, expected answer, or style guidelines. section 7.2.4 section 7.3.5
- Return an inspectable result, such as a pass/fail value and feedback. Criterion-level critique explains what to improve while preserving what worked. section 7.2.4 section 7.3.5
- Use the result to accept, revise, or block the output. If feedback triggers regeneration, cap retries and decide what happens when the limit is reached. section 7.3
Evaluating a retrieval answer
For each benchmark question, a retrieval agent produces a natural-language answer. A second agent receives the question, answer, expected key term, and a wrong-answer example. It returns a typed is_correct boolean and feedback. The key-term test can accept an answer that also contains unsupported extra claims, so add a separate grounding check when claims must be supported by retrieved context. section 7.2.4 section 7.3
Listing 7.5 04_RAG_agent.py (agents only)
agent = Agent(
name="RAG Agent",
instructions="""
You are a retrieval-augmented knowledge agent.
Break down the user's query into smaller parts if needed
to fetch relevant context for the user's query. #1
""",
tools=[search_knowledge_by_keyword],
model="gpt-4o", # Specify the model to use
)
class EvaluationOutput(BaseModel): #2
"""Output model for evaluation agent."""
is_correct: bool
feedback: str
evaluation_agent = Agent( #3
name="Evaluation Agent",
instructions="""
You are an evaluation agent.
Your task is to evaluate the answers provided by the RAG Agent.
You will compare the answer against
the expected answers key term to validate correctness.
""",
output_type=EvaluationOutput,
)
Inspect the answer agent and the evaluator’s typed pass/fail and feedback fields. section 7.2.4
Critiquing a generated image
An image critic assesses output against style guidelines and a five-point rubric. It returns a pass/fail result and feedback that can guide another generation attempt. Criteria such as color, lighting, and consistency make the critique more actionable than a bare pass or fail. section 7.3.5
A rubric only sees what its criteria ask about. Depending on the task, include grounding, completeness, tone, tool use, or error handling—not just surface quality. Rerun tests to check consistency, and track variability, latency, and token use. Maintain curated test sets and repeat judge checks after prompt or model changes. section 7.3.5 section 7.6 section 11.1.5








