Backpropagation through time
Backpropagation through time is a training technique that unrolls recurrent computations across time steps and propagates gradients through the resulting sequence.
Natural Language Processing in Action, Second Edition
Backpropagation through time is a training technique that applies backpropagation across the successive time steps of a recurrent neural network.
- How it is used
- Trains recurrent neural networks for sequential token prediction by carrying error information backward through time steps.
- How deep it goes
- Explains chain-rule gradients, serial backward computation, unrolled shared weights, and batching, without deriving detailed equations.
- Examples
- Uses tokens in sentences or documents, an unrolled RNN graph, and single-sample or batched training passes.
- What you can do after
- Apply it via PyTorch to adjust shared RNN weights from token-level prediction errors.
Sutskever's List
Backpropagation through time is a training technique that unrolls a recurrent computation across time steps and propagates gradients through the resulting sequence.
- How it is used
- Trains recurrent neural networks on sequential predictions by propagating errors across unrolled time steps.
- How deep it goes
- Explains unrolling, shared weights, and vanishing gradients, but stops short of a mathematical derivation.
- Examples
- Uses next-word prediction across a 100-word sequence and the sentence linking “France” to “French.”
- What you can do after
- Evaluate why recurrent models lose learning signals across long sequences and why LSTMs help.
Read how Sutskever's List explains backpropagation through time →
Related concepts
- recurrent neural networkused by · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.