Long-range dependency
A long-range dependency is a relationship between elements separated by substantial distance in a sequence or other structured representation that are jointly relevant.
Sutskever's List
A long-range dependency is a relationship between elements that are separated by a substantial distance in an input sequence or structure.
- How it is used
- Explains how sequence models relate distant elements when processing or generating text.
- How deep it goes
- Compares recurrent information flow with direct self-attention connections, without giving a formal measure of dependency length.
- Examples
- Contrasts character-level text drifting off topic with attention linking “cheese” directly to “mice.”
- What you can do after
- Evaluate how recurrent and attention-based architectures handle distant sequence relationships and computational paths.
Transformers in Action
A long-term dependency is a relationship between sequence elements separated by a substantial distance that influences their interpretation or prediction.
- How it is used
- Motivates transformer attention by identifying a sequence relationship that recurrent processing can struggle to capture.
- How deep it goes
- Explains the architectural contrast and LSTM training difficulty, but does not derive attention calculations or implement a model.
- Examples
- Uses English-to-French translation and a diagram contrasting stepwise RNN connections with direct self-attention connections.
- What you can do after
- Evaluate/understand why self-attention can handle distant sequence relationships without the sequential processing required by RNNs and LSTMs.
Read how Transformers in Action explains long-range dependency →
Learn Generative AI with PyTorch
A relationship between distant elements in a sequence whose interpretation depends on information far apart in the input.
- How it is used
- Guides sequence modeling for text, music, stock prices, and next-token prediction.
- How deep it goes
- Explains hidden-state context and sequence-length tradeoffs through RNN and Transformer training examples.
- Examples
- Uses the sentence “a frog has four legs” and shifted token windows of 100 or 128 positions.
- What you can do after
- Apply it by choosing sequence windows and training next-token predictors that retain earlier context.
Read how Learn Generative AI with PyTorch explains long-range dependency →
Related concepts
- attentionaddressed by · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
- convolutional neural networkcontrasts with · A convolutional neural network is a neural-network architecture that applies learned convolutional operations to extract patterns from structured data.
- transformer architectureaddressed by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
- recurrent neural networkcontrasts with · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
- data representationuses · A data representation is a form or encoding in which information is structured for storage, processing, transmission, or interpretation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.