liveBook · Manning

Annotation artifact

As explained in Evaluating AI Systems

An unintended pattern in evaluation data, arising from its creation or annotation, that can let a system score well without demonstrating the intended capability.

An annotation artifact is an unintended pattern in evaluation data that correlates with a label or answer without reflecting the capability being tested. A system can exploit the pattern and score well while failing to perform the intended task. Artifacts can arise from annotation conventions, templated phrasing, label leakage, or correlations unrelated to the task. section 4.2.1 section 4.5.4

Hypothesis-only natural language inference

Natural language inference asks whether a hypothesis is entailed by, contradicts, or is neutral with respect to a premise. A model given only the hypothesis should perform at chance level if the label depends on the relationship between both sentences. Yet a hypothesis-only model reached about 67% accuracy on SNLI and over 50% on MultiNLI. Words such as “nobody,” “never,” and “no” were strongly associated with contradiction labels, while generic terms such as “animal” and “outdoors” tended to signal entailment. When systems were evaluated only on examples the hypothesis-only model could not solve, their performance dropped significantly. section 4.2.1

A strong benchmark score is therefore not enough to establish the intended capability. The key question is whether a recurring cue is genuinely informative for the decision or merely a by-product of data creation. section 4.2.1 section 4.5.4

To investigate suspected artifacts:

  • Try a trivial or shallow baseline that should not succeed if the task requires the intended reasoning or understanding. Strong performance is a red flag. section 4.5.4
  • Inspect examples the system gets right with very high confidence for repeated phrases, answer structures, verbosity, or other regularities in inputs, outputs, and labels. section 4.5.4
  • Remove, mask, or neutralize suspected incidental cues and rerun the evaluation. A substantial performance drop is strong evidence that the original score was inflated by artifacts. section 4.5.4

How Evaluating AI Systems covers it

How deep it goes
Explains artifact detection through shallow baselines, high-confidence example inspection, and cue removal followed by reevaluation.
Examples
Illustrates artifacts with hypothesis-only predictions on SNLI and MultiNLI and performance drops on harder subsets.
What you can do after
Diagnose potential artifacts by testing shallow baselines, inspecting recurring cues, and reevaluating after neutralizing them.
How it connects
Connects annotation artifacts to spurious correlations, label leakage, benchmark evaluation, and natural language inference.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.