Long-range dependency

As explained in Sutskever's List

A long-range dependency is a relationship between elements that are separated by a substantial distance in an input sequence or structure.

A long-range dependency links elements far apart in a sequence. Character-level models must preserve relevant information across many steps, making it easier to drift off topic or lose coherence. section 4.1

Recurrent neural networks carry information forward step by step. Self-attention instead connects every token directly to every other token at each layer, so distance does not increase the computational path between them. Convolutional models need stacked layers to connect distant positions. section 5.1.1 section 4.1

A distant connection

In the Transformer example, “cheese” can attend directly to “mice,” however many words separate them. Character-level generation faces a longer sequence of steps over which it must preserve coherence. section 5.1.1 section 4.1

Figure 4.1 This image, from Chris Olah’s work (discussed in the next section), shows a chain-like structure representing an RNN. It illustrates how RNNs process data sequentially, with the output of one step feeding into the next. The recurrent, linked modules reflect the RNN’s ability to carry information forward, making it well suited to learning patterns that unfold over time, such as time series, text, or audio. Used with permission granted by the author (Chris Olah).

The RNN chain diagram illustrates sequential information flow, a contrast to direct self-attention connections. section 4.1 section 5.1.1

How Sutskever's List covers it

How deep it goes

Compares recurrent information flow with direct self-attention connections, without giving a formal measure of dependency length.

Examples

Contrasts character-level text drifting off topic with attention linking “cheese” directly to “mice.”

What you can do after

Evaluate how recurrent and attention-based architectures handle distant sequence relationships and computational paths.

How it connects

Connects recurrent neural networks, self-attention, convolutional neural networks, and sequence modeling.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Transformers in ActionA long-term dependency is a relationship between sequence elements separated by a substantial distance that influences their interpretation or prediction.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA relationship between distant elements in a sequence whose interpretation depends on information far apart in the input.

Compare how 3 books explain long-range dependency →

Related concepts

• attentionaddressed by · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• convolutional neural networkcontrasts with · A convolutional neural network is a neural-network architecture that applies learned convolutional operations to extract patterns from structured data.
• transformer architectureaddressed by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• recurrent neural networkcontrasts with · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
• data representationuses · A data representation is a form or encoding in which information is structured for storage, processing, transmission, or interpretation.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.