liveBook · Manning

Rubric

As explained in AI Model Evaluation

A defined set of criteria and scoring guidance for assessing model outputs.

A rubric defines the criteria and scoring guidance used to assess model outputs or compare model candidates. It makes judgments interpretable and repeatable, but it does not eliminate the need to validate the evaluator or consider product impact. section 9.5.2 section 9.3

For model selection, a rubric brings evidence together before an A/B test. Its purpose is not to produce one magic number: it makes trade-offs explicit and records why a candidate should advance, iterate, or wait. section 7.3 section 7.1

A model selection review can compare candidates across several dimensions:

  • Offline performance: recall, precision, fairness, error rate, latency, and inference cost. section 7.3
  • Product heuristics: diversity, novelty, personalization accuracy, and safety constraints. section 7.3
  • Model maturity and strategic alignment: production experience and fit with product or business goals. section 7.3
  • Operational readiness: logging, monitoring hooks, and feature drift detection readiness. section 7.3
  • Sandbox or beta feedback: user-facing quality, known edge cases, and themes from internal testing. section 7.3
Table 7.1 Final review table for Model A vs. Model B
Category
Metric/Criteria
Model A
Model B
Notes
Offline performance
Recall@10
0.82
.78
 
Offline performance
Precision@10
.81
.79
 
Offline performance
Fairness gap (abs diff)
3%
1%
Model B is more balanced
Offline performance
Latency (p95 ms)
380ms
430ms
Model A is faster; however, this reflects prior tuning and optimization.
Offline performance
Cost per 1M inferences
$22
$24
 
Product heuristics
Diversity index
.72
.78
Model B surfaces more varied titles
 
Product heuristics
Novelty (% unseen content)
28%
33%
 
Model maturity
Current production usage
Yes
No
Model B’s latency degradation relative to Model A will be addressed in future iterations.
Operational readiness
Logging & monitoring hooks in place
Yes
Yes
 
Sandbox / Internal beta testing
Positive/neutral/negative ratio
80/15/5
70/20/10
Model B feedback flagged some personalization misses

Compare two recommendation models across performance, product, readiness, and feedback criteria. The table makes a faster model's advantage visible alongside the other model's better fairness and diversity. section 7.3

For an LLM judge, the rubric is part of the evaluation prompt: it specifies what to inspect and how to score it. Define terms such as factuality, helpfulness, and safety rather than asking the judge to infer what “good” means. Score separate dimensions before requesting an overall decision, and specify a machine-readable output schema when results must be aggregated. section 9.5.2

Evaluate a support chatbot

Ask the judge to score correctness, completeness, tone, and policy compliance separately, then return an overall pass/fail judgment. Provide the context needed for those judgments and define the scoring rules and output fields. section 9.5.2

Before writing the rubric, clarify the decision, choose representative inputs and reference context, and decide whether the result should be a label, score, or ranking. For example, compare summaries on clarity, factuality, and redundancy while providing the original source. A vague request to rate “quality” can produce arbitrary results. section 9.4

Test the judge against examples with known judgments, inspect disagreements, and validate it against human ratings, gold examples, or known decision criteria. A consistent judge can still favor long or polished answers, show position bias, or miss factual errors. Fixed prompts and rubrics support comparison across versions, but offline scores alone do not capture the full live product experience. section 9.3 section 9.5.2 section 1.4

For each evaluation run, record the rubric, allowed scores or labels, prompt and judge versions, sampling strategy, dataset snapshot, and calibration results. If scores shift unexpectedly, check data and judge changes as well as rubric wording and output schema; even a minor rubric edit can change judge behavior and require re-validation. section 9.10.2 section 10.6.2

How AI Model Evaluation covers it

How deep it goes
Explains review dimensions, judge-prompt design, validation, documentation, and score-change triage without prescribing a universal scoring formula.
Examples
Compares recommendation models across fairness, latency, diversity, and beta feedback, and scores a support chatbot on separate quality dimensions.
What you can do after
Apply a rubric to compare model candidates or specify and validate an LLM judge's scoring criteria.
How it connects
Connects offline evaluation, A/B testing, product heuristics, human judgment, and LLM-as-a-judge.

Also explained in

Compare how 7 books explain rubric →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.