A robust evaluation metric gives consistent, meaningful results when minor, irrelevant factors change. A fragile metric can swing sharply because of noise or superficial differences, making results less trustworthy and progress harder to track. section 3.3.5
When small changes distort scores
BLEU may drop when a synonym or punctuation mark changes despite equivalent meaning. Perplexity can shift with tokenization or preprocessing, while human ratings can vary with wording or presentation order. section 3.3.5 section 3.3.5
Check whether a metric remains stable by:
- Testing controlled variations that should not affect measurement, such as paraphrased patient questions or changed response formatting. section 3.3.5
- Repeating evaluation across datasets, domains, or environments to see whether results hold. section 3.3.5
- Resampling evaluation data to estimate how much the metric varies across samples. section 3.3.5











