liveBook · Manning

Long-range dependency

As explained in Build an Advanced RAG Application (From Scratch)

A long-range dependency is a relationship between sequence elements separated by many intervening positions.

A long-range dependency connects sequence elements separated by many intervening positions. Capturing it requires using information from faraway parts of a sequence when interpreting later elements. section 2.1

RNNs and GRUs process sequences step by step, carrying earlier information through a hidden state. Over long or complex sequences, that information can become difficult to preserve, alongside vanishing or exploding gradients during training. section 2.0 section 2.1

LSTMs improve handling of long-range dependencies through internal memory. Transformers take a different approach: self-attention processes the sentence in parallel and identifies relevant relationships among distant words. section 2.0 section 2.1

A sentence with distant relationships

In “The quick brown fox jumps over the lazy dog,” an RNN or GRU carries context sequentially, while a Transformer compares relationships across the whole sentence to identify the most relevant words. section 2.1

Figure 2.1 The strength of the transformer model is in its ability to understand the significance and context of every word in a sentence, unlike RNNs which primarily assess the connection of a word with its immediate neighbor.

Use the figure to compare immediate-neighbor processing in RNNs with a Transformer's broader attention to word significance and context. section 2.1

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines the problem and compares recurrent memory with self-attention, but stops short of deriving either mechanism mathematically.
Examples
Uses a familiar fox sentence plus dialogue, speech recognition, translation, and text-generation contexts.
What you can do after
Understand why Transformers capture distant sequence relationships more effectively than sequential RNN and GRU processing.
How it connects
Connects long-range dependencies to RNNs, LSTMs, GRUs, Transformers, self-attention, and vanishing or exploding gradients.

Also explained in

Compare how 6 books explain long-range dependency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.