A recurrent neural network processes sequence elements one at a time while maintaining a hidden state that carries information from earlier inputs. This sequential design supports sequence modeling but limits parallel processing and can weaken context over long sequences. section 2.1 section 2.5
A recurrent neural network operates through a repeated sequence of steps:
- Reads one sequence element at a time rather than processing the entire sequence in parallel. section 2.1
- Updates a hidden state that carries information from previous elements into later processing. section 2.1
- May struggle with long-range dependencies and experience vanishing or exploding gradients during training. section 2.1
Processing a sentence
For “The quick brown fox jumps over the lazy dog,” an RNN processes each word sequentially and carries earlier information through its hidden state. Compared with a Transformer, it has less direct access to distant word relationships. section 2.1
Figure 2.1 The strength of the transformer model is in its ability to understand the significance and context of every word in a sentence, unlike RNNs which primarily assess the connection of a word with its immediate neighbor.
Compare recurrent processing with self-attention: the figure emphasizes that RNNs primarily assess connections with immediate neighbors, while Transformers evaluate broader word significance and context. section 2.1

















