Noise sensitivity measures whether a model still behaves reasonably when realistic imperfections alter its inputs. In recommendation systems, missing history, masked genres, altered popularity scores, missing ratings, or mislabeled metadata can change ranked recommendations. section 3.2.3
Test it by comparing the model’s clean-input output with its output after realistic perturbations. For ranked recommendations, define output difference explicitly rather than treating it as a single prediction change. section 3.2.3
Useful comparison measures include:
- Top-K overlap: count how many items remain in the top 10. section 3.2.3
- Ranking stability: compare whether items retain roughly the same order. section 3.2.3
- Performance degradation: measure drops in precision@K, recall@K, or NDCG. section 3.2.3
- Exposure shift: check whether noise changes blockbuster, long-tail, or diverse-content recommendations. section 3.2.3
Use more than one metric: a model can change substantially while remaining relevant, or remain stable while recommending low-quality items. section 3.2.3
Figure 3.10 illustrates the formula for noise sensitivity, which measures how much a model’s output changes when noise is introduced to the input data. The numerator represents the difference between the model output on noisy input and the model output on clean input. The denominator normalizes that difference against the original output, making the result easier to compare across examples or model versions.
The formula normalizes the difference between noisy-input and clean-input outputs against the original output, supporting comparisons across examples or model versions. section 3.2.3
