Backpropagation through time trains a recurrent neural network by unrolling its loop across time steps. The resulting sequence resembles a deep linear network whose weights are shared across steps, allowing errors from later predictions to update earlier computations. section 4.2
Unrolling also exposes a central limitation: a 100-word prediction resembles a 100-layer network. Gradients linking the first and last words can shrink exponentially, leaving little learning signal for long-range dependencies. LSTM networks address this difficulty with gated memory rather than changing the basic need to train across time. section 4.2
Remembering “France”
To predict “French” in “I grew up in France … I speak fluent French,” the network must carry information across intervening words. Backpropagation through time sends the prediction error backward across those steps, but the signal may become negligibly small. section 4.2
