liveBook · Manning

Long-range dependency

As explained in Machines that Think

A relationship between sequence elements separated by many intervening positions.

A long-range dependency is a relationship between sequence elements separated by many intervening positions. In language, an earlier detail can shape the interpretation of a much later word or phrase, sometimes across sentences or extended discourse. section 2.6.2

These relationships appear in several familiar forms:

  • A subject introduced early in a sentence can govern agreement many words later. section 2.6.2
  • A pronoun can refer back to an earlier idea, while a conditional can link events separated by entire phrases. section 2.6.2

Models focused on nearby context can capture local patterns while losing track of earlier details. The result may be a brittle interpretation or output that drifts over time. Word embeddings capture similarity, but provide little structure for maintaining continuity across extended discourse. section 2.6.2

Earlier neural architectures faced different versions of this problem. Convolutional models emphasize local patterns; recurrent models carry information forward in an evolving state, where signals can fade and earlier interpretations become difficult to revise. Gated recurrent models extend memory but do not eliminate that difficulty. section 3.5.1 section 3.6

Attention offers a different approach: it lets a model weigh and retrieve relevant information from positions throughout a sequence. Transformers make attention central, allowing tokens to interact directly rather than relying on a strictly sequential state. This makes distant relationships easier to capture, though continuity remains a challenge in real-world language use. section 3.6 section 4.1.1 section 2.6.2

How Machines that Think covers it

How deep it goes
Explains linguistic forms and contrasts architectural mechanisms and limitations, without detailing an implementation.
Examples
Uses subject agreement, pronoun reference, and conditionals to illustrate relationships across intervening language.
What you can do after
Evaluate or understand how model architectures differ in their treatment of distant context and continuity.
How it connects
Connects dependencies to attention, Transformers, recurrent networks, convolutional networks, and embeddings.

Also explained in

Compare how 6 books explain long-range dependency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.