liveBook · Manning

Evaluation criterion

As explained in AI Model Evaluation

A primary metric, or success metric, is the main outcome measure selected to evaluate whether an experiment achieves its intended objective.

An evaluation criterion defines what outcome would confirm an experiment’s hypothesis. Set it before the test so the result can guide a clear decision: roll out, iterate, or continue testing. section 6.3.3 section 6.6

A primary metric is only part of the evaluation contract. Pair it with guardrails and explicit trade-offs so a gain in one measure does not conceal harm to broader product outcomes. section 8.1.2

Movie recommendations

If the goal is to increase interaction with promotional titles, use click-through rate (CTR) as the primary metric. Track completion rate, retention, skips, or thumbs-downs as guardrails; specify how much consumption may decline, if at all. section 8.1.2

When results are ambiguous, return to the original criterion and decision framework. A weak or noisy result may mean the experiment could not detect the effect, not that the model had no effect. section 6.6

How AI Model Evaluation covers it

How deep it goes
Explains primary metrics alongside guardrails and explicit trade-offs, including a non-inferiority margin example.
Examples
Uses movie-recommender CTR as a primary metric, with completion, retention, skips, and thumbs-downs as guardrails.
What you can do after
Evaluate whether to launch, iterate, or retest by comparing the primary metric with guardrails and the original decision framework.
How it connects
Links hypotheses, guardrail metrics, decision frameworks, and product strategy in A/B testing.

Also explained in

Compare how 4 books explain evaluation criterion →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.