A rubric is a scoring guide that states what good agent behavior looks like and how to judge it. It gives human evaluators and models a common standard, especially when an open-ended answer has no simple ground truth. A plausible final answer is not enough: evaluation may also need to examine tool choices and the path that produced it. section 10.2 section 10.0
Build the rubric from observed failures rather than impressions:
- Collect real execution traces. Note what went wrong, preferably starting with the first error; for example, an agent may claim a failed tool call succeeded. section 10.2
- Group similar errors into a failure taxonomy, then assess their frequency and severity. section 10.2
- Turn the important failure patterns into explicit criteria and define metrics that make the resulting judgments trackable. section 10.2 section 10.2.4
Choose a decision rule for each criterion. A graded scale can express degrees of quality, but PASS/FAIL is generally easier to apply consistently: evaluators need not defend the difference between adjacent scores. For a research answer, useful checks are answer relevancy, citation reliability, and requirement compliance. A metric is separate from the rubric: overall pass rate, for example, counts cases that pass all three checks and divides by the total evaluated. section 10.2.4
A research-answer rubric
Relevancy: PASS if the response answers the user’s core question; FAIL if it merely supplies plausible background. Citation reliability: PASS if key claims are supported and citations point to that support; FAIL if citations are missing or mismatched. Requirement compliance: PASS if the requested format and scope are followed; FAIL if required elements are omitted. These separate decisions reveal which kind of failure an overall result hides. section 10.2.4
To automate those decisions with an LLM judge, supply the user query, final answer, rubric, and relevant source excerpts with their metadata; include an expected answer when available. Instruct the judge to use only the supplied evidence, mark unsupported claims and citation mismatches as failures, and return a single JSON result. This keeps the judgment tied to the stated criteria rather than the judge’s outside knowledge or preference for polished prose. Code-based checks, human review, and end-user feedback can complement LLM judging. section 10.3.2 section 10.3
A rubric is not permanent. As users, operating conditions, and tools change, review new traces, revise the failure taxonomy and criteria, and update the test set and metrics. Otherwise, a stable score can conceal failures the rubric no longer captures. section 10.4.2
