An evaluation criterion defines what outcome would confirm an experiment’s hypothesis. Set it before the test so the result can guide a clear decision: roll out, iterate, or continue testing. section 6.3.3 section 6.6
A primary metric is only part of the evaluation contract. Pair it with guardrails and explicit trade-offs so a gain in one measure does not conceal harm to broader product outcomes. section 8.1.2
Movie recommendations
If the goal is to increase interaction with promotional titles, use click-through rate (CTR) as the primary metric. Track completion rate, retention, skips, or thumbs-downs as guardrails; specify how much consumption may decline, if at all. section 8.1.2
When results are ambiguous, return to the original criterion and decision framework. A weak or noisy result may mean the experiment could not detect the effect, not that the model had no effect. section 6.6



