A spurious correlation links a feature to a label without reflecting the relationship a system is meant to learn. In natural language inference benchmarks, annotation artifacts let models exploit superficial cues instead of reasoning about sentence meaning. section 4.2.1
Hypothesis-only prediction
In SNLI and MultiNLI, a model using only the hypothesis achieved unexpectedly high accuracy. Words such as “nobody,” “never,” and “no” often signaled contradiction, while generic terms such as “animal” and “outdoors” often signaled entailment. Performance dropped when NLI systems were tested on examples the hypothesis-only model could not solve, revealing how full-benchmark scores could reward superficial cues. section 4.2.1
Table 4.2 Common limitations and risks of benchmark-based evaluation
|
Limitation/risk
|
Description
|
|
Representation
gap
|
Benchmarks
are often cleaner, narrower, and more structured than real deployment
environments, which can make performance appear better than it actually is.
|
|
Staleness
and saturation
|
Benchmarks
age over time and may become too easy, making it difficult to distinguish
meaningful improvements.
|
|
Narrow
optimization
|
Teams may
optimize for benchmark scores rather than real-world performance, leading to
brittle systems that exploit dataset quirks.
|
|
Hidden
biases
|
Benchmark
datasets may underrepresent populations, scenarios, or perspectives,
concealing important performance disparities.
|
Benchmark quirks can encourage narrow optimization and brittle systems that perform well by exploiting dataset patterns. section 4.2.1

