A rubric defines the criteria and scoring guidance used to assess model outputs or compare model candidates. It makes judgments interpretable and repeatable, but it does not eliminate the need to validate the evaluator or consider product impact. section 9.5.2 section 9.3
For model selection, a rubric brings evidence together before an A/B test. Its purpose is not to produce one magic number: it makes trade-offs explicit and records why a candidate should advance, iterate, or wait. section 7.3 section 7.1
A model selection review can compare candidates across several dimensions:
- Offline performance: recall, precision, fairness, error rate, latency, and inference cost. section 7.3
- Product heuristics: diversity, novelty, personalization accuracy, and safety constraints. section 7.3
- Model maturity and strategic alignment: production experience and fit with product or business goals. section 7.3
- Operational readiness: logging, monitoring hooks, and feature drift detection readiness. section 7.3
- Sandbox or beta feedback: user-facing quality, known edge cases, and themes from internal testing. section 7.3
Table 7.1 Final review table for Model A vs. Model B
|
Category
|
Metric/Criteria
|
Model A
|
Model B
|
Notes
|
|
Offline performance
|
Recall@10
|
0.82
|
.78
|
|
|
Offline performance
|
Precision@10
|
.81
|
.79
|
|
|
Offline performance
|
Fairness gap (abs diff)
|
3%
|
1%
|
Model B is more balanced
|
|
Offline performance
|
Latency (p95 ms)
|
380ms
|
430ms
|
Model A is faster; however, this reflects prior tuning and optimization.
|
|
Offline performance
|
Cost per 1M inferences
|
$22
|
$24
|
|
|
Product heuristics
|
Diversity index
|
.72
|
.78
|
Model B surfaces more varied titles
|
|
Product heuristics
|
Novelty (% unseen content)
|
28%
|
33%
|
|
|
Model maturity
|
Current production usage
|
Yes
|
No
|
Model B’s latency degradation relative to Model A will be addressed in future iterations.
|
|
Operational readiness
|
Logging & monitoring hooks in place
|
Yes
|
Yes
|
|
|
Sandbox / Internal beta testing
|
Positive/neutral/negative ratio
|
80/15/5
|
70/20/10
|
Model B feedback flagged some personalization misses
|
Compare two recommendation models across performance, product, readiness, and feedback criteria. The table makes a faster model's advantage visible alongside the other model's better fairness and diversity. section 7.3
For an LLM judge, the rubric is part of the evaluation prompt: it specifies what to inspect and how to score it. Define terms such as factuality, helpfulness, and safety rather than asking the judge to infer what “good” means. Score separate dimensions before requesting an overall decision, and specify a machine-readable output schema when results must be aggregated. section 9.5.2
Evaluate a support chatbot
Ask the judge to score correctness, completeness, tone, and policy compliance separately, then return an overall pass/fail judgment. Provide the context needed for those judgments and define the scoring rules and output fields. section 9.5.2
Before writing the rubric, clarify the decision, choose representative inputs and reference context, and decide whether the result should be a label, score, or ranking. For example, compare summaries on clarity, factuality, and redundancy while providing the original source. A vague request to rate “quality” can produce arbitrary results. section 9.4
Test the judge against examples with known judgments, inspect disagreements, and validate it against human ratings, gold examples, or known decision criteria. A consistent judge can still favor long or polished answers, show position bias, or miss factual errors. Fixed prompts and rubrics support comparison across versions, but offline scores alone do not capture the full live product experience. section 9.3 section 9.5.2 section 1.4
For each evaluation run, record the rubric, allowed scores or labels, prompt and judge versions, sampling strategy, dataset snapshot, and calibration results. If scores shift unexpectedly, check data and judge changes as well as rubric wording and output schema; even a minor rubric edit can change judge behavior and require re-validation. section 9.10.2 section 10.6.2






