Long-range dependency
A long-range dependency is a relationship between elements separated by substantial distance in a sequence or other structured representation that are jointly relevant.
Sutskever's List
A long-range dependency is a relationship between elements that are separated by a substantial distance in an input sequence or structure.
- How it is used
- Explains how sequence models relate distant elements when processing or generating text.
- How deep it goes
- Compares recurrent information flow with direct self-attention connections, without giving a formal measure of dependency length.
- Examples
- Contrasts character-level text drifting off topic with attention linking “cheese” directly to “mice.”
- What you can do after
- Evaluate how recurrent and attention-based architectures handle distant sequence relationships and computational paths.
Transformers in Action
A long-term dependency is a relationship between sequence elements separated by a substantial distance that influences their interpretation or prediction.
- How it is used
- Motivates attention as an alternative to recurrent processing for relationships across sequences.
- How deep it goes
- Compares recurrent and attention-based approaches and discusses training stability, without deriving attention calculations.
- Examples
- Illustrates contextual relationships with English-to-French translation and the sentiment interaction between “not” and “bad.”
- What you can do after
- Evaluate how recurrent and attention-based architectures differ in handling dependencies across a sequence.
Read how Transformers in Action explains long-range dependency →
Learn Generative AI with PyTorch
A relationship between distant elements in a sequence whose interpretation depends on information far apart in the input.
- How it is used
- Guides sequence modeling for text, music, stock prices, and next-token prediction.
- How deep it goes
- Explains hidden-state context and sequence-length tradeoffs through RNN and Transformer training examples.
- Examples
- Uses the sentence “a frog has four legs” and shifted token windows of 100 or 128 positions.
- What you can do after
- Apply it by choosing sequence windows and training next-token predictors that retain earlier context.
Read how Learn Generative AI with PyTorch explains long-range dependency →
Related concepts
- attentionaddressed by · Attention is a neural-network mechanism that computes input-dependent relevance weights among representations and uses them to combine information for a context-dependent computation.
- convolutional neural networkcontrasts with · A convolutional neural network is a neural network that applies learned convolutional operations to structured data to detect local patterns and learn hierarchical representations.
- recurrent neural networkcontrasts with · A recurrent neural network is a neural network that processes ordered inputs by carrying and updating an internal state, allowing information from earlier inputs to influence later computations.
- transformer architectureaddressed by · A transformer is a neural-network architecture that models relationships among input elements primarily through attention and related transformations.
- data representationuses · A data representation is a form or encoding that structures information for interpretation, processing, storage, or exchange.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


