Rubric
A rubric is a structured set of explicit criteria and guidance for evaluating and assigning judgments or scores to an output, artifact, or behavior.
Build an AI Agent (From Scratch)
An explicit framework of criteria and performance standards for judging the quality or success of an output.
- How it is used
- Gives human evaluators and LLM judges explicit criteria for assessing open-ended agent behavior.
- How deep it goes
- Develops criteria from trace review and failure taxonomy, then specifies PASS/FAIL decisions, aggregate metrics, and an LLM-judge prompt.
- Examples
- Uses PASS/FAIL checks for answer relevancy, citation reliability, and requirement compliance in a research-agent evaluation prompt.
- What you can do after
- Build it from scratch by converting observed failures into criteria and using them to grade responses and track pass rates.
AI Agents in Action, Second Edition
A rubric is a structured set of criteria and standards for evaluating the quality of an output or behavior.
- How it is used
- Evaluates open-ended agent outputs against explicit quality criteria when a single correct answer or accuracy score is insufficient.
- How deep it goes
- Develops objectives, criteria, scales, thresholds, human validation, and a critic-agent workflow rather than stopping at a definition.
- Examples
- Uses recommendation criteria and an image-style critic with a five-point scale, pass threshold, and typed feedback.
- What you can do after
- Build it from scratch for an open-ended output, then use a critic agent to score it and guide revision.
Read how AI Agents in Action, Second Edition explains rubric →
AI Agents in Action
A rubric is a structured set of criteria used to specify or evaluate the quality of an agent profile or response.
- How it is used
- Guides profile and prompt evaluation when accuracy alone cannot capture use-case-specific quality.
- How deep it goes
- Details objectives, criteria, scales, descriptions, scoring, and automated evaluation, but does not specify a universal rubric.
- Examples
- Scores recommendations by genre, subject, and format using a one-to-five alignment scale.
- What you can do after
- Apply it via Prompt flow to score recommendations automatically or establish a manual evaluation baseline.
Reinforcement Learning from Human Feedback
A rubric is a structured set of weighted evaluation criteria for judging the quality of a response to a specific task or prompt.
- How it is used
- Uses prompt-specific criteria as rewards for improving multiple generated answers through reinforcement learning.
- How deep it goes
- Defines hard rules and principles, demonstrates them on a prompt, and specifies a weighted configuration schema.
- Examples
- Uses a museum exhibit request requiring five culturally and historically distinct artifacts with significance, mystery, and engaging presentation.
- What you can do after
- Apply it via reinforcement learning to evaluate and improve responses without clearly verifiable answers.
Read how Reinforcement Learning from Human Feedback explains rubric →
Related concepts
- evaluatorused by · An evaluator is a model or component that assesses outputs against specified criteria and produces judgments, scores, comparisons, or feedback.
- groundingused by · Grounding is the practice of anchoring or constraining an AI system’s outputs with relevant information, context, or evidence.
- objective functionproduces · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
- inter-rater reliabilitymeasures · Inter-rater reliability is the degree of agreement or consistency among independent raters evaluating the same items, subjects, or events.
- evaluationused by · Evaluation is the systematic assessment of a system, result, or approach against relevant criteria, evidence, or objectives.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.