A long-range dependency connects sequence elements separated by many intervening positions. Capturing it requires using information from faraway parts of a sequence when interpreting later elements. section 2.1
RNNs and GRUs process sequences step by step, carrying earlier information through a hidden state. Over long or complex sequences, that information can become difficult to preserve, alongside vanishing or exploding gradients during training. section 2.0 section 2.1
LSTMs improve handling of long-range dependencies through internal memory. Transformers take a different approach: self-attention processes the sentence in parallel and identifies relevant relationships among distant words. section 2.0 section 2.1
A sentence with distant relationships
In “The quick brown fox jumps over the lazy dog,” an RNN or GRU carries context sequentially, while a Transformer compares relationships across the whole sentence to identify the most relevant words. section 2.1
Figure 2.1 The strength of the transformer model is in its ability to understand the significance and context of every word in a sentence, unlike RNNs which primarily assess the connection of a word with its immediate neighbor.
Use the figure to compare immediate-neighbor processing in RNNs with a Transformer's broader attention to word significance and context. section 2.1





