An n-gram model predicts from a fixed window: after “hel,” it can use only the sequences it has counted within that window. A recurrent neural network (RNN) instead carries a hidden state forward. At each step, it combines the current input with the previous state to update an internal summary of what came before. The state—not the step’s output—is passed to the next step. section 4.1
What should follow “hell”?
When reading “hello” character by character, an RNN updates its state after each letter. By the time it reaches “hell,” its prediction can draw on the context carried through those updates rather than treating the next letter as an isolated choice. section 4.1
Carrying state does not guarantee remembering everything. To predict “French” in “I grew up in France … I speak fluent French,” an RNN needs information from many words earlier. As that gap grows, training signals to earlier steps can become vanishingly small. LSTMs were developed to better handle such long-range dependencies; their cells use learned gates and distinguish the hidden state from a separate cell state. section 4.2
Figure 4.2 A single LSTM cell. The inputs are the current vector Xt, the previous hidden state ht–1, and the previous cell state Ct–1. There are three learned gates: the forget ft, input it, and output Ot. They are all produced by sigmoid layers (“σ” blocks). A tanh layer forms a candidate update \(\tilde{C}\)t. Pointwise multiplications (“×”) and an addition (“+”) update the cell state on the top “conveyor” pathway. Arrows indicate vector flow; the legend marks pointwise operations, concatenation, and copies. This is the standard LSTM, emphasizing how gating enables selective memory retention and exposure. Used with permission granted by the author (Chris Olah), though it is slightly modified from its original form.
The LSTM cell diagram separates the previous hidden state from the previous cell state and shows the gates that control the cell-state update. section 4.2
The idea also applies beyond sequences. In a message-passing neural network, each atom in a molecular graph maintains a hidden state vector. It updates that vector from messages sent by neighboring atoms; after several rounds, the state summarizes information from its neighborhood. A readout then pools atom states to predict a molecular property. section 7.3.2
A final hidden state can also become a bottleneck. In early RNN encoder–decoder translation, one fixed-length vector had to carry the source sentence’s relevant information to the decoder. Longer sentences could lose detail. Bahdanau attention addressed this by letting the decoder draw on different input positions as it generated the translation, rather than relying on that single summary. section 5.3
