Rubric
A rubric is a structured set of explicit criteria and guidance for judging the quality of an output, artifact, or behavior.
AI Model Evaluation
A defined set of criteria and scoring guidance for assessing model outputs.
- How it is used
- Organizes model-selection evidence before A/B testing and defines scoring criteria for LLM judges.
- How deep it goes
- Explains review dimensions, judge-prompt design, validation, documentation, and score-change triage without prescribing a universal scoring formula.
- Examples
- Compares recommendation models across fairness, latency, diversity, and beta feedback, and scores a support chatbot on separate quality dimensions.
- What you can do after
- Apply a rubric to compare model candidates or specify and validate an LLM judge's scoring criteria.
Build an AI Agent (From Scratch)
An explicit framework of criteria and performance standards for judging the quality or success of an output.
- How it is used
- Defines shared standards for judging open-ended agent behavior and supporting automated evaluation.
- How deep it goes
- Traces rubric design from execution review and failure taxonomy through binary criteria and aggregated metrics.
- Examples
- Uses answer relevancy, citation reliability, and requirement compliance to grade a research agent’s response.
- What you can do after
- Practice deriving pass/fail criteria from observed failures and using them to score agent outputs.
LLM Evaluation and Alignment, The Foundational Ideas
An evaluation rubric is a structured set of explicit criteria and guidance for assigning judgments to an AI system’s outputs.
- How it is used
- Defines evaluation criteria that guide LLM judges, human labelers, and preference-labeling workflows.
- How deep it goes
- Descends from quality definitions to score-level criteria, calibration examples, edge-case guidance, and ongoing reliability monitoring.
- Examples
- Uses helpfulness score levels, coherence criteria for summaries, representative response examples, and ambiguous middle cases.
- What you can do after
- Evaluate open-ended outputs with reproducible criteria and calibrate judges against human-labeled comparisons.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains rubric →
AI Agents in Action, Second Edition
A rubric is a structured set of criteria and standards for evaluating the quality of an output or behavior.
- How it is used
- Evaluates open-ended agent outputs where quality has multiple dimensions and a single accuracy measure is inadequate.
- How deep it goes
- Defines objectives and measurable criteria, then demonstrates a scoring scale, pass threshold, and critic-based feedback loop.
- Examples
- Uses image style guidelines, a five-point scale, and a pass/fail image critique to illustrate a rubric.
- What you can do after
- Apply it via the Agents SDK using a critic that returns a typed pass/fail result and feedback.
Read how AI Agents in Action, Second Edition explains rubric →
AI Agents in Action
A rubric is a structured set of criteria used to specify or evaluate the quality of an agent profile or response.
- How it is used
- Guides evaluation of agent profiles and recommendations against criteria suited to the desired outcome.
- How deep it goes
- Walks through objectives, criteria, scales, descriptions, scoring, and optional score aggregation.
- Examples
- Scores recommended titles separately for subject, format, and genre using a five-point alignment scale.
- What you can do after
- Apply it via prompt flow to score recommendations and compare profile variants.
Reinforcement Learning from Human Feedback
A rubric is a structured set of weighted evaluation criteria for judging the quality of a response to a specific task or prompt.
- How it is used
- Uses prompt-specific criteria as rewards for improving multiple generated answers through reinforcement learning.
- How deep it goes
- Defines hard rules and principles, demonstrates them on a prompt, and specifies a weighted configuration schema.
- Examples
- Uses a museum exhibit request requiring five culturally and historically distinct artifacts with significance, mystery, and engaging presentation.
- What you can do after
- Apply it via reinforcement learning to evaluate and improve responses without clearly verifiable answers.
Read how Reinforcement Learning from Human Feedback explains rubric →
Self-Improving Agents
A set of criteria for evaluating an output.
- How it is used
- Defines the standard an LLM judge uses to measure output quality.
- How deep it goes
- Defines rubrics as plain-language judge prompts and illustrates them with task-specific criteria.
- Examples
- Uses research-answer criteria and a support-task rubric focused on direct yes-or-no answers.
- What you can do after
- Evaluate competing outputs against task-specific criteria and record a winner with brief feedback.
Also covered in
- Evaluating AI SystemsIn early access (MEAP)
- LLM Customization and Fine-TuningIn early access (MEAP)
What to know first
- AI agentAn AI agent is a computational system that uses artificial intelligence to interpret information and select or perform actions toward goals while interacting with an environment.
- human-in-the-loopHuman-in-the-loop is a system or workflow design in which people participate in reviewing, guiding, correcting, or authorizing an automated process.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- promptA prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
Related concepts
- LLM-as-a-judgeused by · LLM-as-a-judge is an evaluation technique in which a language model assesses or compares outputs according to specified criteria.
- A/B testuses · An A/B test is a controlled experiment that compares outcomes for two variants assigned to different users or experimental units.
- checklistcontrasts with · A structured list of tasks, questions, or criteria used to guide an activity or systematically verify that relevant requirements have been addressed.
- evaluationused by · Evaluation is the systematic assessment of a system, model, process, or outcome against relevant criteria or objectives using evidence.
- judge promptpart of · A judge prompt is an instruction given to an evaluating language model that defines how it should assess another model's output.
- groundingused by · Grounding is the practice of anchoring or constraining an AI system’s outputs with relevant information, context, or evidence.
Questions about rubric
What is rubric?
A rubric is a structured set of explicit criteria and guidance for judging the quality of an output, artifact, or behavior.
Which Manning books explain rubric?
AI Model Evaluation, Build an AI Agent (From Scratch), LLM Evaluation and Alignment, The Foundational Ideas, AI Agents in Action, Second Edition, AI Agents in Action, Reinforcement Learning from Human Feedback and Self-Improving Agents.
What should I know before learning rubric?
Manning books that teach rubric usually introduce AI agent, human-in-the-loop, language model and prompt first.
What is the difference between rubric and checklist?
A rubric is a structured set of explicit criteria and guidance for judging the quality of an output, artifact, or behavior. A structured list of tasks, questions, or criteria used to guide an activity or systematically verify that relevant requirements have been addressed.
What is rubric part of?
Rubric is part of judge prompt. A judge prompt is an instruction given to an evaluating language model that defines how it should assess another model's output.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.






