liveBook · Manning

Evaluation criterion

Also called success criteria, success metric, Evaluation Criteria, success criterion, decision criterion

A defined condition, standard, or measure used to assess an outcome, option, system, or decision.

How 4 Manning books explain it

Think Like a CTO

An interview criterion is a defined quality, capability, outcome, or standard used to evaluate a candidate for a role.

How it is used
Structures interviews around role-specific qualities and provides a common basis for candidate comparison and discussion.
How deep it goes
Defines criteria, score ranges, and scorecards, but stops short of prescribing a universal hiring decision.
Examples
Uses articulation, professionalism, software development lifecycle, design patterns, coding knowledge, and AWS experience as examples.
What you can do after
Apply it via a role-specific scorecard with defined evidence ranges and documented interviewer scores.

Read how Think Like a CTO explains evaluation criterion →

Evaluating AI Systems

A stated condition or standard used to determine whether a system satisfies a decision requirement.

How it is used
Guides evaluation choices toward concrete decisions about whether an AI system is fit for its intended use.
How deep it goes
Distinguishes dimensions, metrics, and thresholds, and explains how to select criteria and choose between binary and scalar judgments.
Examples
Uses legal and medical chatbots to illustrate task scope, deployment thresholds, contraindication checks, and graded judgments.
What you can do after
Apply evaluation criteria to scope a decision and choose suitable judgments for assessing system behavior.

Read how Evaluating AI Systems explains evaluation criterion →

AI Model Evaluation

A primary metric, or success metric, is the main outcome measure selected to evaluate whether an experiment achieves its intended objective.

How it is used
Defines what outcome would confirm an experiment’s hypothesis and guide rollout or further iteration.
How deep it goes
Explains primary metrics alongside guardrails and explicit trade-offs, including a non-inferiority margin example.
Examples
Uses movie-recommender CTR as a primary metric, with completion, retention, skips, and thumbs-downs as guardrails.
What you can do after
Evaluate whether to launch, iterate, or retest by comparing the primary metric with guardrails and the original decision framework.

Read how AI Model Evaluation explains evaluation criterion →

LLM Evaluation and Alignment, The Foundational Ideas

An evaluation criterion is a defined standard against which an AI system’s behavior is judged.

How it is used
Defines what good looks like before measuring and aligning an AI system.
How deep it goes
Defines criteria, demonstrates measured scores, and connects failures to corrective interventions.
Examples
Uses a customer-service chatbot evaluated for accuracy, politeness, latency, and issue resolution.
What you can do after
Evaluate/understand it by defining task-specific standards, measuring performance, diagnosing failures, and guiding alignment changes.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains evaluation criterion →

Also covered in

What to know first

Related concepts

Questions about evaluation criterion

What is evaluation criterion?

A defined condition, standard, or measure used to assess an outcome, option, system, or decision.

Which Manning books explain evaluation criterion?

Think Like a CTO, Evaluating AI Systems, AI Model Evaluation and LLM Evaluation and Alignment, The Foundational Ideas.

What should I know before learning evaluation criterion?

Manning books that teach evaluation criterion usually introduce language model, accuracy, fine-tuning and metric first.

What is the difference between evaluation criterion and metric?

A defined condition, standard, or measure used to assess an outcome, option, system, or decision. A metric is a quantitative measure that assigns values to a property or outcome so it can be assessed, compared, or monitored.

What kinds of evaluation criterion are there?

Kinds of evaluation criterion include relevance.

What is evaluation criterion part of?

Evaluation criterion is part of prompt structure. Prompt structure is the organization of a prompt’s content and instructions to guide a model’s response or generation.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.