A recurrent neural network (RNN) processes an ordered sequence one step at a time. At each step, it updates a hidden state that carries information from earlier inputs. This makes it useful for text, speech, and time series: unlike a feed-forward network that treats inputs independently, an RNN can use what came before to help predict what comes next. An n-gram model also uses prior context, but only within a fixed window; an RNN carries a representation of the sequence history. section 4.0 section 4.2
Figure 4.1 This image, from Chris Olah’s work (discussed in the next section), shows a chain-like structure representing an RNN. It illustrates how RNNs process data sequentially, with the output of one step feeding into the next. The recurrent, linked modules reflect the RNN’s ability to carry information forward, making it well suited to learning patterns that unfold over time, such as time series, text, or audio. Used with permission granted by the author (Chris Olah).
The linked modules show the central idea: each step passes information forward to the next as the network reads a sequence. section 4.1
Remembering a distant clue
In “I grew up in France … I speak fluent French,” predicting “French” depends on remembering “France” across the intervening words. The hidden state can, in principle, carry that clue. In practice, ordinary RNNs often lose useful information as the gap grows. Training unrolls the recurrence across time, so learning a relationship across 100 words resembles training through 100 layers: gradients conveying the earlier word’s importance can become negligibly small. section 4.2
Long short-term memory (LSTM) networks are a specialized RNN designed to retain relevant information over longer spans. This addressed a core weakness, but memory was not the only obstacle. As researchers tried to scale recurrent models, overfitting held back language-model performance. In 2014, Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals introduced selective dropout tailored to LSTMs. Their engineering intervention helped turn a promising sequence model into a more effective one. section 4.2 section 4.3
Generating text one character at a time
Andrej Karpathy’s 2015 character-level model learned from roughly one million characters of Paul Graham’s essays. A two-layer LSTM predicted and sampled successive characters; a temperature setting controlled whether output was repetitive or more adventurous. The result picked up startup vocabulary and stylistic patterns but did not produce reliable advice. It made both the appeal and the limits of recurrent text generation easy to see. section 4.1
RNNs also powered end-to-end speech recognition, including Deep Speech 2. In early neural machine translation, an RNN encoder compressed a source sentence into its final hidden state for a decoder. That fixed-length representation could lose details from longer sentences. Bahdanau attention let the decoder consult different encoder states instead; reversing source sentences was another, more architecture-dependent way to shorten the path between relevant inputs and early outputs. Transformers later replaced recurrence with parallelizable attention, changing the practical trade-off for sequence modeling. section 4.0 section 5.3 section 5.7 section 1.3
