Long-range dependency

As explained in Transformers in Action

A long-term dependency is a relationship between sequence elements separated by a substantial distance that influences their interpretation or prediction.

A long-range dependency is a relationship between sequence elements far enough apart that earlier information can affect how a later element is interpreted or predicted. Capturing such relationships is a challenge for sequence models: an RNN processes elements one at a time, updating a recurrent state, and can struggle with long sequences and distant relationships. section 1.1.1 section 2.1.2

An LSTM can selectively remember or forget earlier information, and attention can help it account for long-term dependencies. But its input still arrives sequentially, slowing training and inference. Stacking recurrent layers can represent more complex patterns, yet makes error signals harder to propagate during training and can cause stability problems such as vanishing gradients. section 2.1.2 section 2.1.1

Transformer self-attention offers a different route: each position can consider other positions directly rather than relying on a state passed from one time step to the next. Attention scores express contextual relationships across the sequence; multihead attention lets the model examine different relationships in parallel. This supports long-range relationships without sequential input processing or deep recurrent-layer stacking. section 1.1.2 section 2.1.2

Figure 2.2 Comparing RNN (left) and self-attention (right) architectures. As illustrated, transformers, with their use of self-attention, can directly attend to all positions in the sequence, regardless of their position in time. Dashed lines represent the flow of information or dependencies. The variables x₁ to x₅ represent data input, and the circles above represent the state of the network at each timestep for the RNN. In contrast, for the self-attention mechanism, the circles above denote how each state considers information from all other states, not just the immediate predecessor.

Compare the RNN’s step-to-step information flow with self-attention’s direct links between sequence positions. section 2.1.2

Translation as a sequence task

To translate “I don’t speak French” into “Je ne parle pas français,” an RNN encoder reads the input word by word and passes a final context vector to its decoder. A transformer instead uses attention to relate input words to the rest of the sentence while producing contextual features for translation. The example illustrates the architectural difference; it does not isolate a particular distant word pair. section 1.1.1 section 1.1.2

How Transformers in Action covers it

How deep it goes

Explains the architectural contrast and LSTM training difficulty, but does not derive attention calculations or implement a model.

Examples

Uses English-to-French translation and a diagram contrasting stepwise RNN connections with direct self-attention connections.

What it assumes

Introduces recurrent-state processing and transformer attention through translation rather than assuming their implementation details.

What you can do after

Evaluate/understand why self-attention can handle distant sequence relationships without the sequential processing required by RNNs and LSTMs.

How it connects

Connects long-range dependencies to recurrent states, LSTMs, vanishing gradients, self-attention, and multihead attention.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListA long-range dependency is a relationship between elements that are separated by a substantial distance in an input sequence or structure.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA relationship between distant elements in a sequence whose interpretation depends on information far apart in the input.

Compare how 3 books explain long-range dependency →

Related concepts

• attentionaddressed by · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• convolutional neural networkcontrasts with · A convolutional neural network is a neural-network architecture that applies learned convolutional operations to extract patterns from structured data.
• transformer architectureaddressed by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• recurrent neural networkcontrasts with · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
• data representationuses · A data representation is a form or encoding in which information is structured for storage, processing, transmission, or interpretation.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.