liveBook · Manning

Rubric

A rubric is a structured set of explicit criteria and guidance for judging the quality of an output, artifact, or behavior.

How 7 Manning books explain it

AI Model Evaluation

A defined set of criteria and scoring guidance for assessing model outputs.

How it is used
Organizes model-selection evidence before A/B testing and defines scoring criteria for LLM judges.
How deep it goes
Explains review dimensions, judge-prompt design, validation, documentation, and score-change triage without prescribing a universal scoring formula.
Examples
Compares recommendation models across fairness, latency, diversity, and beta feedback, and scores a support chatbot on separate quality dimensions.
What you can do after
Apply a rubric to compare model candidates or specify and validate an LLM judge's scoring criteria.

Read how AI Model Evaluation explains rubric →

Build an AI Agent (From Scratch)

An explicit framework of criteria and performance standards for judging the quality or success of an output.

How it is used
Defines shared standards for judging open-ended agent behavior and supporting automated evaluation.
How deep it goes
Traces rubric design from execution review and failure taxonomy through binary criteria and aggregated metrics.
Examples
Uses answer relevancy, citation reliability, and requirement compliance to grade a research agent’s response.
What you can do after
Practice deriving pass/fail criteria from observed failures and using them to score agent outputs.

Read how Build an AI Agent (From Scratch) explains rubric →

LLM Evaluation and Alignment, The Foundational Ideas

An evaluation rubric is a structured set of explicit criteria and guidance for assigning judgments to an AI system’s outputs.

How it is used
Defines evaluation criteria that guide LLM judges, human labelers, and preference-labeling workflows.
How deep it goes
Descends from quality definitions to score-level criteria, calibration examples, edge-case guidance, and ongoing reliability monitoring.
Examples
Uses helpfulness score levels, coherence criteria for summaries, representative response examples, and ambiguous middle cases.
What you can do after
Evaluate open-ended outputs with reproducible criteria and calibrate judges against human-labeled comparisons.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains rubric →

AI Agents in Action, Second Edition

A rubric is a structured set of criteria and standards for evaluating the quality of an output or behavior.

How it is used
Evaluates open-ended agent outputs where quality has multiple dimensions and a single accuracy measure is inadequate.
How deep it goes
Defines objectives and measurable criteria, then demonstrates a scoring scale, pass threshold, and critic-based feedback loop.
Examples
Uses image style guidelines, a five-point scale, and a pass/fail image critique to illustrate a rubric.
What you can do after
Apply it via the Agents SDK using a critic that returns a typed pass/fail result and feedback.

Read how AI Agents in Action, Second Edition explains rubric →

AI Agents in Action

A rubric is a structured set of criteria used to specify or evaluate the quality of an agent profile or response.

How it is used
Guides evaluation of agent profiles and recommendations against criteria suited to the desired outcome.
How deep it goes
Walks through objectives, criteria, scales, descriptions, scoring, and optional score aggregation.
Examples
Scores recommended titles separately for subject, format, and genre using a five-point alignment scale.
What you can do after
Apply it via prompt flow to score recommendations and compare profile variants.

Read how AI Agents in Action explains rubric →

Reinforcement Learning from Human Feedback

A rubric is a structured set of weighted evaluation criteria for judging the quality of a response to a specific task or prompt.

How it is used
Uses prompt-specific criteria as rewards for improving multiple generated answers through reinforcement learning.
How deep it goes
Defines hard rules and principles, demonstrates them on a prompt, and specifies a weighted configuration schema.
Examples
Uses a museum exhibit request requiring five culturally and historically distinct artifacts with significance, mystery, and engaging presentation.
What you can do after
Apply it via reinforcement learning to evaluate and improve responses without clearly verifiable answers.

Read how Reinforcement Learning from Human Feedback explains rubric →

Self-Improving Agents

A set of criteria for evaluating an output.

How it is used
Defines the standard an LLM judge uses to measure output quality.
How deep it goes
Defines rubrics as plain-language judge prompts and illustrates them with task-specific criteria.
Examples
Uses research-answer criteria and a support-task rubric focused on direct yes-or-no answers.
What you can do after
Evaluate competing outputs against task-specific criteria and record a winner with brief feedback.

Read how Self-Improving Agents explains rubric →

Also covered in

What to know first

Related concepts

Questions about rubric

What is rubric?

A rubric is a structured set of explicit criteria and guidance for judging the quality of an output, artifact, or behavior.

Which Manning books explain rubric?

AI Model Evaluation, Build an AI Agent (From Scratch), LLM Evaluation and Alignment, The Foundational Ideas, AI Agents in Action, Second Edition, AI Agents in Action, Reinforcement Learning from Human Feedback and Self-Improving Agents.

What should I know before learning rubric?

Manning books that teach rubric usually introduce AI agent, human-in-the-loop, language model and prompt first.

What is the difference between rubric and checklist?

A rubric is a structured set of explicit criteria and guidance for judging the quality of an output, artifact, or behavior. A structured list of tasks, questions, or criteria used to guide an activity or systematically verify that relevant requirements have been addressed.

What is rubric part of?

Rubric is part of judge prompt. A judge prompt is an instruction given to an evaluating language model that defines how it should assess another model's output.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.