liveBook · Manning

Evaluation contract

As explained in AI Model Evaluation

An evaluation contract is a formal specification of what is judged, which criteria apply, what context is available, and what output the evaluator must produce.

An evaluation contract specifies what success means and what must be protected. For an LLM judge, it defines the judging task, criteria, available context, and required output. For an online model test, it connects product strategy to a primary metric, guardrails, and acceptable trade-offs. section 9.5.1 section 8.1.2

Define a judge’s task by answering four questions:

  • What is being judged? Identify the artifact, such as an answer, recommendation list, or pair of outputs. section 9.5.1
  • Which criteria apply? Choose qualities relevant to the decision, such as factual accuracy, relevance, or safety. section 9.5.1
  • What context is available? Specify the query, source documents, policy, or other evidence the judge needs. section 9.5.1
  • What output is required? Request a score, label, winner, explanation, or structured object. section 9.5.1

Bound a factuality judgment

Instead of asking “Judge this answer,” ask whether it is supported by the provided source, sufficiently answers the user’s question, and avoids unsupported claims. Specify the required output as well. The bounded source and explicit criteria give the judge a job that can be validated later. section 9.5.1

Protect viewing satisfaction

A movie recommender test might seek higher click-through rate while monitoring completion, retention, skips, or thumbs-downs. An explicit non-inferiority margin could require consumption to be no more than 2% worse than control, so a click-through gain alone does not count as success. section 8.1.2

Contracts should also account for effects that emerge over time. A model that promotes popular titles may initially increase clicks while reducing catalog diversity and retention for some users. Randomized exposure or exploration traffic, long-term guardrails, and tracking offline–online metric correlations help evaluate that feedback loop. section 8.2

How AI Model Evaluation covers it

How deep it goes
Specifies four judge-task questions and details primary metrics, guardrails, and non-inferiority margins for online tests.
Examples
Uses a source-bounded factuality judgment and a movie recommender test balancing click-through rate against consumption.
What you can do after
Evaluate and understand model quality by defining explicit judging criteria and online success conditions with guardrails.
How it connects
Connects LLM-as-a-judge, A/B testing, feedback loops, and non-inferiority margins.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.