Evaluation criterion
A defined condition, standard, or measure used to assess an outcome, option, system, or decision.
Think Like a CTO
An interview criterion is a defined quality, capability, outcome, or standard used to evaluate a candidate for a role.
- How it is used
- Structures interviews around role-specific qualities and provides a common basis for candidate comparison and discussion.
- How deep it goes
- Defines criteria, score ranges, and scorecards, but stops short of prescribing a universal hiring decision.
- Examples
- Uses articulation, professionalism, software development lifecycle, design patterns, coding knowledge, and AWS experience as examples.
- What you can do after
- Apply it via a role-specific scorecard with defined evidence ranges and documented interviewer scores.
Evaluating AI Systems
A stated condition or standard used to determine whether a system satisfies a decision requirement.
- How it is used
- Guides evaluation choices toward concrete decisions about whether an AI system is fit for its intended use.
- How deep it goes
- Distinguishes dimensions, metrics, and thresholds, and explains how to select criteria and choose between binary and scalar judgments.
- Examples
- Uses legal and medical chatbots to illustrate task scope, deployment thresholds, contraindication checks, and graded judgments.
- What you can do after
- Apply evaluation criteria to scope a decision and choose suitable judgments for assessing system behavior.
Read how Evaluating AI Systems explains evaluation criterion →
AI Model Evaluation
A primary metric, or success metric, is the main outcome measure selected to evaluate whether an experiment achieves its intended objective.
- How it is used
- Defines what outcome would confirm an experiment’s hypothesis and guide rollout or further iteration.
- How deep it goes
- Explains primary metrics alongside guardrails and explicit trade-offs, including a non-inferiority margin example.
- Examples
- Uses movie-recommender CTR as a primary metric, with completion, retention, skips, and thumbs-downs as guardrails.
- What you can do after
- Evaluate whether to launch, iterate, or retest by comparing the primary metric with guardrails and the original decision framework.
Read how AI Model Evaluation explains evaluation criterion →
LLM Evaluation and Alignment, The Foundational Ideas
An evaluation criterion is a defined standard against which an AI system’s behavior is judged.
- How it is used
- Defines what good looks like before measuring and aligning an AI system.
- How deep it goes
- Defines criteria, demonstrates measured scores, and connects failures to corrective interventions.
- Examples
- Uses a customer-service chatbot evaluated for accuracy, politeness, latency, and issue resolution.
- What you can do after
- Evaluate/understand it by defining task-specific standards, measuring performance, diagnosing failures, and guiding alignment changes.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains evaluation criterion →
Also covered in
- Think Like a Software Engineering Manager
- The Architecture of ReasoningIn early access (MEAP)
- Generative AI in Action
- Grokking Software ArchitectureIn early access (MEAP)
- Effective Conversational AI
- Effective Platform Engineering
- Statistics Every Programmer Needs
- Optimization Algorithms: AI techniques for design, planning, and control problems
- Build an AI Agent (From Scratch)
- API Design Patterns
- Financial AI in Practice
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- metricA metric is a quantitative measure that assigns values to a property or outcome so it can be assessed, compared, or monitored.
Related concepts
- prompt structurepart of · Prompt structure is the organization of a prompt’s content and instructions to guide a model’s response or generation.
- metriccontrasts with · A metric is a quantitative measure that assigns values to a property or outcome so it can be assessed, compared, or monitored.
- A/B testpart of · An A/B test is a controlled experiment that compares outcomes for two variants assigned to different users or experimental units.
- relevanceincludes · Relevance is the degree to which information, a result, or a response pertains to a query, task, purpose, or information need.
- decision treepart of · A decision tree is a tree-structured model or representation in which successive tests or choices lead through branches to outcomes, predictions, or actions.
Questions about evaluation criterion
What is evaluation criterion?
A defined condition, standard, or measure used to assess an outcome, option, system, or decision.
Which Manning books explain evaluation criterion?
Think Like a CTO, Evaluating AI Systems, AI Model Evaluation and LLM Evaluation and Alignment, The Foundational Ideas.
What should I know before learning evaluation criterion?
Manning books that teach evaluation criterion usually introduce language model, accuracy, fine-tuning and metric first.
What is the difference between evaluation criterion and metric?
A defined condition, standard, or measure used to assess an outcome, option, system, or decision. A metric is a quantitative measure that assigns values to a property or outcome so it can be assessed, compared, or monitored.
What kinds of evaluation criterion are there?
Kinds of evaluation criterion include relevance.
What is evaluation criterion part of?
Evaluation criterion is part of prompt structure. Prompt structure is the organization of a prompt’s content and instructions to guide a model’s response or generation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



