liveBook · Manning

Noise sensitivity

As explained in AI Model Evaluation

Noise sensitivity measures how much a model's outputs change when realistic noise or perturbations are introduced into its inputs.

Noise sensitivity measures whether a model still behaves reasonably when realistic imperfections alter its inputs. In recommendation systems, missing history, masked genres, altered popularity scores, missing ratings, or mislabeled metadata can change ranked recommendations. section 3.2.3

Test it by comparing the model’s clean-input output with its output after realistic perturbations. For ranked recommendations, define output difference explicitly rather than treating it as a single prediction change. section 3.2.3

Useful comparison measures include:

  • Top-K overlap: count how many items remain in the top 10. section 3.2.3
  • Ranking stability: compare whether items retain roughly the same order. section 3.2.3
  • Performance degradation: measure drops in precision@K, recall@K, or NDCG. section 3.2.3
  • Exposure shift: check whether noise changes blockbuster, long-tail, or diverse-content recommendations. section 3.2.3

Use more than one metric: a model can change substantially while remaining relevant, or remain stable while recommending low-quality items. section 3.2.3

Figure 3.10 illustrates the formula for noise sensitivity, which measures how much a model’s output changes when noise is introduced to the input data. The numerator represents the difference between the model output on noisy input and the model output on clean input. The denominator normalizes that difference against the original output, making the result easier to compare across examples or model versions.

The formula normalizes the difference between noisy-input and clean-input outputs against the original output, supporting comparisons across examples or model versions. section 3.2.3

How AI Model Evaluation covers it

How deep it goes
Defines perturbation comparisons and several output measures, but leaves the choice of difference metric contextual.
Examples
Tests missing watch-history items, masked genres, altered popularity scores, missing ratings, and mislabeled metadata.
What you can do after
Evaluate model stability and recommendation quality under realistic noisy inputs using multiple complementary measures.
How it connects
Connects robustness testing with ranked lists, precision@K, recall@K, NDCG, and exposure shift.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.