liveBook · Manning

Rubric

As explained in Self-Improving Agents

A set of criteria for evaluating an output.

A rubric tells an LLM judge what “good” means for a task. Written in plain language as the judge’s system prompt, it sets the standard behind the evaluation signal. section 2.2.3 section 2.2

Comparing research answers

A pairwise judge can prioritize factual accuracy against a reference, grounding in retrieved sources, and clarity. It returns a winner—LEFT, RIGHT, or TIE—and brief feedback. section 2.2.3

Improving a support answer

For a customer’s yes-or-no question, a one-sentence rubric asks how directly the answer addresses it. The judge can use that signal to prefer a prompt that puts the verdict first. section 1.2.1

Figure 2.5 details the internal process of how LLM-as-Judge uses pairwise evaluation to compare multiple outputs against the same rubric without having position bias modify the answer.

Trace how the same rubric is applied to both answer orders, filtering out verdicts that change with position. section 2.2.3

How Self-Improving Agents covers it

How deep it goes
Defines rubrics as plain-language judge prompts and illustrates them with task-specific criteria.
Examples
Uses research-answer criteria and a support-task rubric focused on direct yes-or-no answers.
What you can do after
Evaluate competing outputs against task-specific criteria and record a winner with brief feedback.
How it connects
Connects rubrics to LLM-as-Judge, pairwise evaluation, and evaluation signals.

Also explained in

Compare how 7 books explain rubric →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Self-Improving Agents. The section links point to the parts of the book it draws on.