liveBook · Manning

Signal

As explained in AI Model Evaluation

A signal is the portion of an observed evaluation result that reflects the real product effect or outcome of interest.

Signal is the part of an evaluation result that reflects the real product effect you care about. In online evaluation, that might mean improved retention, more satisfying sessions, or higher-quality engagement. Noise comes from random variation such as sampling quirks, day-of-week effects, novelty, or unrelated behavior. section 8.1.1

A metric becomes useful only when it matches the product goal. Weak metrics can turn attention into an apparent win and redirect evaluation toward the wrong definition of success. section 8.1.1

Movie recommendations

For a streaming movie recommender, a click-through-rate increase may be noise with respect to the real outcome if it produces clickbait titles without improving consumption, satisfaction, or retention. For a promotional advertising system whose goal is clicks, the same metric can be a coherent signal. section 8.1.1

Treat an observed result as credible signal only after checking the evaluation system:

  • Validate that the measure reflects the intended quality or product outcome. section 10.4.1
  • Test stability with repeated sampling and swapped evaluation order; inconsistent scores can overwhelm a real difference between variants. section 10.4.1
  • When results change unexpectedly, check data, prompts, rubrics, schemas, and judge variance before declaring a model regression. section 10.6.2

How AI Model Evaluation covers it

How deep it goes
Distinguishes product effects from common noise sources and adds validation, stability, and triage checks.
Examples
Uses movie-recommender CTR, sustained viewing satisfaction, judge score gaps, pass-rate gates, and canary-set changes.
What it assumes
Assumes familiarity with product metrics and evaluation results while clarifying the distinction between signal and noise.
What you can do after
Diagnose/debug with it by separating genuine product changes from weak metrics, unstable judges, and evaluation regressions.
How it connects
Connects online evaluation, metrics, noise, retention, judge stability, and model regression.

Also explained in

Compare how 14 books explain signal →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.