A recurrent neural network (RNN) processes a sequence one element at a time. Unlike a network that treats inputs independently, it carries a hidden state forward through a loop. That state summarizes earlier inputs, letting the current prediction depend on what came before. This makes RNNs useful for ordered data such as text, music, and stock prices. section 8.1.2
Predicting the next word
Consider “a frog has four legs.” After reading “a,” an RNN predicts the next word. At the next step, it uses “frog” and the hidden state carried from the previous step. By the time it predicts “legs,” its state reflects the preceding words. Order matters: swapping the teams in “Kentucky defeated Vanderbilt” reverses who won, even though the words are otherwise unchanged. section 8.1.2 section 8.1.1
Figure 8.1 How a layer of recurrent neurons unfolds through time. When a recurrent neural network makes a prediction on sequential data, it takes the hidden state from the previous time step, h(t – 1), along with the input at the current time step, x(t), and generates the output, y(t), and the updated hidden state, h(t). The hidden state at time step t captures the information in all previous time steps, x(0), x(1), …, x(t).
The unfolded diagram shows the repeated step: current input x(t) and previous hidden state h(t − 1) produce an output y(t) and updated state h(t). section 8.1.2
For a text generator, connect the recurrent model to the surrounding text pipeline:
- Tokenize the text into characters, words, or subwords, then represent words numerically with embeddings. section 8.6
- Build and train an LSTM, an improved RNN variant, to generate text. section 8.0 section 8.1
- Adjust generated text with temperature or top-K sampling: these change how predictable or varied the next-word choices are. section 8.6
The hidden state helps preserve context, but a standard RNN struggles with long-range dependencies because of vanishing gradients. LSTMs handle longer ranges better, though even they can falter over extremely long distances. RNNs also process positions sequentially, limiting training parallelism. Transformers offer a contrasting approach: self-attention captures distant relationships while allowing parallel training. section 8.1.2 section 1.3
