A recurrent neural network (RNN) processes ordered inputs one step at a time. At each step, it combines the current input with a hidden state carrying information from earlier steps. That state supplies context for later predictions. section 12.1.1
The hidden state has fixed size, so each update must retain, modify, or discard information. As sequences grow, distant relationships can weaken; for example, the sentence “The book that the professor who the students admired assigned was difficult to understand” requires preserving the link between “book” and “was.” Computation is sequential too: each state depends on the preceding one, limiting parallel training. section 12.1.1 section 12.0
Long short-term memory and gated recurrent units were developed to preserve information more effectively, but the basic state-propagation constraint remains. Transformers offer another approach: self-attention models relationships among tokens directly, with positional encodings preserving order. This architectural contrast does not make RNNs universally obsolete, and an RNN is distinct from Bellman’s recurrence relations for sequential optimization. section 12.1.1 section 12.3.2

















