Long-range dependency

As explained in Learn Generative AI with PyTorch

A relationship between distant elements in a sequence whose interpretation depends on information far apart in the input.

A long-range dependency links distant elements in a sequence when interpreting one element requires information from far earlier in the input. In language modeling, the next-word prediction may depend on the entire preceding sequence rather than only the current word. section 8.1.2 section 8.1.1

Sequence models handle this requirement by carrying context forward, but the choice of model and training window matters:

• An RNN uses a hidden state containing a summary of earlier inputs, allowing each prediction to use the current input together with accumulated sequence information. section 8.1.2
• Standard RNNs handle short-term dependencies more easily than longer-range dependencies, so preserving distant context remains a central challenge. section 8.1.2
• Training sequences that are too short can prevent the model from learning long-range dependencies, while longer sequences can slow training. section 8.3.2 section 12.2.2

Predicting the next word

For “a frog has four legs,” the prediction of “legs” should use “a frog has four,” not just “four.” RNN processing carries this earlier context through successive hidden states. section 8.1.2

Figure 8.1 How a layer of recurrent neurons unfolds through time. When a recurrent neural network makes a prediction on sequential data, it takes the hidden state from the previous time step, h(t – 1), along with the input at the current time step, x(t), and generates the output, y(t), and the updated hidden state, h(t). The hidden state at time step t captures the information in all previous time steps, x(0), x(1), …, x(t).

The unfolded recurrent layer shows how each step combines the current input x(t) with the previous hidden state h(t−1), producing an output and updated state. section 8.1.2

How Learn Generative AI with PyTorch covers it

How deep it goes

Explains hidden-state context and sequence-length tradeoffs through RNN and Transformer training examples.

Examples

Uses the sentence “a frog has four legs” and shifted token windows of 100 or 128 positions.

What it assumes

Assumes basic sequence modeling while introducing token windows, hidden states, embeddings, and next-token targets.

What you can do after

Apply it by choosing sequence windows and training next-token predictors that retain earlier context.

How it connects

Connects recurrent neural networks, hidden states, language modeling, sequence-to-sequence prediction, and Transformers.

liveBookmanning.com

Read Learn Generative AI with PyTorch on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListA long-range dependency is a relationship between elements that are separated by a substantial distance in an input sequence or structure.
• 

  liveBookmanning.com
  Transformers in ActionA long-term dependency is a relationship between sequence elements separated by a substantial distance that influences their interpretation or prediction.

Compare how 3 books explain long-range dependency →

Related concepts

• attentionaddressed by · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• convolutional neural networkcontrasts with · A convolutional neural network is a neural-network architecture that applies learned convolutional operations to extract patterns from structured data.
• transformer architectureaddressed by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• recurrent neural networkcontrasts with · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
• data representationuses · A data representation is a form or encoding in which information is structured for storage, processing, transmission, or interpretation.

This explainer was generated by Manning's knowledge graph from the text of Learn Generative AI with PyTorch. The section links point to the parts of the book it draws on.