liveBook · Manning

Spurious correlation

As explained in Evaluating AI Systems

A correlation between data features and labels that does not reflect the underlying relationship a system is meant to learn.

A spurious correlation links a feature to a label without reflecting the relationship a system is meant to learn. In natural language inference benchmarks, annotation artifacts let models exploit superficial cues instead of reasoning about sentence meaning. section 4.2.1

Hypothesis-only prediction

In SNLI and MultiNLI, a model using only the hypothesis achieved unexpectedly high accuracy. Words such as “nobody,” “never,” and “no” often signaled contradiction, while generic terms such as “animal” and “outdoors” often signaled entailment. Performance dropped when NLI systems were tested on examples the hypothesis-only model could not solve, revealing how full-benchmark scores could reward superficial cues. section 4.2.1

Table 4.2 Common limitations and risks of benchmark-based evaluation
Limitation/risk
Description
Representation gap
Benchmarks are often cleaner, narrower, and more structured than real deployment environments, which can make performance appear better than it actually is.
Staleness and saturation
Benchmarks age over time and may become too easy, making it difficult to distinguish meaningful improvements.
Narrow optimization
Teams may optimize for benchmark scores rather than real-world performance, leading to brittle systems that exploit dataset quirks.
Hidden biases
Benchmark datasets may underrepresent populations, scenarios, or perspectives, concealing important performance disparities.

Benchmark quirks can encourage narrow optimization and brittle systems that perform well by exploiting dataset patterns. section 4.2.1

How Evaluating AI Systems covers it

How deep it goes
Explains annotation cues and a hard-subset evaluation, without detailing how the artifacts were introduced.
Examples
Uses hypothesis-only prediction, label-associated words, and evaluation on examples the model could not solve.
What you can do after
Evaluate benchmark results by checking whether systems succeed beyond examples solvable through superficial cues.
How it connects
Connects spurious correlations to annotation artifacts, natural language inference, and benchmark evaluation.

Also explained in

Compare how 2 books explain spurious correlation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.