liveBook · Manning

Long-range dependency

Also called long-term dependencies

A long-range dependency is a relationship between elements separated by substantial distance in a sequence or other structured representation that are jointly relevant.

How 6 Manning books explain it

Build AI Drug Discovery Pipelines

A long-range interaction is a biologically relevant relationship between sequence positions that are far apart in the primary sequence but may be spatially close in the folded structure.

How it is used
Frames distant sequence relationships as a central modeling problem in molecular generation and protein structure prediction.
How deep it goes
Explains biological and SMILES examples, recurrent failure from vanishing gradients, GRU gates, and transformer attention.
Examples
Uses benzene and complex SMILES parsing, distant protein contacts, beta sheets, and disulfide bonds as concrete examples.
What you can do after
Evaluate model architectures by whether they can preserve or directly access information across distant sequence positions.

Read how Build AI Drug Discovery Pipelines explains long-range dependency →

Sutskever's List

A long-range dependency is a relationship between elements that are separated by a substantial distance in an input sequence or structure.

How it is used
Explains how sequence models relate distant elements when processing or generating text.
How deep it goes
Compares recurrent information flow with direct self-attention connections, without giving a formal measure of dependency length.
Examples
Contrasts character-level text drifting off topic with attention linking “cheese” directly to “mice.”
What you can do after
Evaluate how recurrent and attention-based architectures handle distant sequence relationships and computational paths.

Read how Sutskever's List explains long-range dependency →

Build an Advanced RAG Application (From Scratch)

A long-range dependency is a relationship between sequence elements separated by many intervening positions.

How it is used
Explains why sequence models must preserve information across distant positions for language understanding and generation.
How deep it goes
Defines the problem and compares recurrent memory with self-attention, but stops short of deriving either mechanism mathematically.
Examples
Uses a familiar fox sentence plus dialogue, speech recognition, translation, and text-generation contexts.
What you can do after
Understand why Transformers capture distant sequence relationships more effectively than sequential RNN and GRU processing.

Read how Build an Advanced RAG Application (From Scratch) explains long-range dependency →

Machines that Think

A relationship between sequence elements separated by many intervening positions.

How it is used
Frames distant relationships as a central challenge in representing language and choosing neural architectures.
How deep it goes
Explains linguistic forms and contrasts architectural mechanisms and limitations, without detailing an implementation.
Examples
Uses subject agreement, pronoun reference, and conditionals to illustrate relationships across intervening language.
What you can do after
Evaluate or understand how model architectures differ in their treatment of distant context and continuity.

Read how Machines that Think explains long-range dependency →

Transformers in Action

A long-term dependency is a relationship between sequence elements separated by a substantial distance that influences their interpretation or prediction.

How it is used
Motivates attention as an alternative to recurrent processing for relationships across sequences.
How deep it goes
Compares recurrent and attention-based approaches and discusses training stability, without deriving attention calculations.
Examples
Illustrates contextual relationships with English-to-French translation and the sentiment interaction between “not” and “bad.”
What you can do after
Evaluate how recurrent and attention-based architectures differ in handling dependencies across a sequence.

Read how Transformers in Action explains long-range dependency →

Learn Generative AI with PyTorch

A relationship between distant elements in a sequence whose interpretation depends on information far apart in the input.

How it is used
Guides sequence modeling for text, music, stock prices, and next-token prediction.
How deep it goes
Explains hidden-state context and sequence-length tradeoffs through RNN and Transformer training examples.
Examples
Uses the sentence “a frog has four legs” and shifted token windows of 100 or 128 positions.
What you can do after
Apply it by choosing sequence windows and training next-token predictors that retain earlier context.

Read how Learn Generative AI with PyTorch explains long-range dependency →

Also covered in

What to know first

Related concepts

Questions about long-range dependency

What is long-range dependency?

A long-range dependency is a relationship between elements separated by substantial distance in a sequence or other structured representation that are jointly relevant.

Which Manning books explain long-range dependency?

Build AI Drug Discovery Pipelines, Sutskever's List, Build an Advanced RAG Application (From Scratch), Machines that Think, Transformers in Action and Learn Generative AI with PyTorch.

What should I know before learning long-range dependency?

Manning books that teach long-range dependency usually introduce language model, recurrent neural network, natural language processing and context first.

What is the difference between long-range dependency and convolutional neural network?

A long-range dependency is a relationship between elements separated by substantial distance in a sequence or other structured representation that are jointly relevant. A convolutional neural network is a neural-network architecture that processes structured inputs with convolutional operations, typically using learned local filters to capture patterns.

What is long-range dependency also called?

Long-range dependency is also called long-term dependencies.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.