A long-range dependency is a relationship between sequence elements separated by many intervening positions. In language, an earlier detail can shape the interpretation of a much later word or phrase, sometimes across sentences or extended discourse. section 2.6.2
These relationships appear in several familiar forms:
- A subject introduced early in a sentence can govern agreement many words later. section 2.6.2
- A pronoun can refer back to an earlier idea, while a conditional can link events separated by entire phrases. section 2.6.2
Models focused on nearby context can capture local patterns while losing track of earlier details. The result may be a brittle interpretation or output that drifts over time. Word embeddings capture similarity, but provide little structure for maintaining continuity across extended discourse. section 2.6.2
Earlier neural architectures faced different versions of this problem. Convolutional models emphasize local patterns; recurrent models carry information forward in an evolving state, where signals can fade and earlier interpretations become difficult to revise. Gated recurrent models extend memory but do not eliminate that difficulty. section 3.5.1 section 3.6
Attention offers a different approach: it lets a model weigh and retrieve relevant information from positions throughout a sequence. Transformers make attention central, allowing tokens to interact directly rather than relying on a strictly sequential state. This makes distant relationships easier to capture, though continuity remains a challenge in real-world language use. section 3.6 section 4.1.1 section 2.6.2





