Signal is the part of an evaluation result that reflects the real product effect you care about. In online evaluation, that might mean improved retention, more satisfying sessions, or higher-quality engagement. Noise comes from random variation such as sampling quirks, day-of-week effects, novelty, or unrelated behavior. section 8.1.1
A metric becomes useful only when it matches the product goal. Weak metrics can turn attention into an apparent win and redirect evaluation toward the wrong definition of success. section 8.1.1
Movie recommendations
For a streaming movie recommender, a click-through-rate increase may be noise with respect to the real outcome if it produces clickbait titles without improving consumption, satisfaction, or retention. For a promotional advertising system whose goal is clicks, the same metric can be a coherent signal. section 8.1.1
Treat an observed result as credible signal only after checking the evaluation system:
- Validate that the measure reflects the intended quality or product outcome. section 10.4.1
- Test stability with repeated sampling and swapped evaluation order; inconsistent scores can overwhelm a real difference between variants. section 10.4.1
- When results change unexpectedly, check data, prompts, rubrics, schemas, and judge variance before declaring a model regression. section 10.6.2













