An evaluation contract specifies what success means and what must be protected. For an LLM judge, it defines the judging task, criteria, available context, and required output. For an online model test, it connects product strategy to a primary metric, guardrails, and acceptable trade-offs. section 9.5.1 section 8.1.2
Define a judge’s task by answering four questions:
- What is being judged? Identify the artifact, such as an answer, recommendation list, or pair of outputs. section 9.5.1
- Which criteria apply? Choose qualities relevant to the decision, such as factual accuracy, relevance, or safety. section 9.5.1
- What context is available? Specify the query, source documents, policy, or other evidence the judge needs. section 9.5.1
- What output is required? Request a score, label, winner, explanation, or structured object. section 9.5.1
Bound a factuality judgment
Instead of asking “Judge this answer,” ask whether it is supported by the provided source, sufficiently answers the user’s question, and avoids unsupported claims. Specify the required output as well. The bounded source and explicit criteria give the judge a job that can be validated later. section 9.5.1
Protect viewing satisfaction
A movie recommender test might seek higher click-through rate while monitoring completion, retention, skips, or thumbs-downs. An explicit non-inferiority margin could require consumption to be no more than 2% worse than control, so a click-through gain alone does not count as success. section 8.1.2
Contracts should also account for effects that emerge over time. A model that promotes popular titles may initially increase clicks while reducing catalog diversity and retention for some users. Randomized exposure or exploration traffic, long-term guardrails, and tracking offline–online metric correlations help evaluate that feedback loop. section 8.2
