Backpropagation through time

As explained in Natural Language Processing in Action, Second Edition

Backpropagation through time is a training technique that applies backpropagation across the successive time steps of a recurrent neural network.

Backpropagation through time trains an RNN by propagating prediction error backward across the sequence of tokens. During the forward pass, gradients are calculated; each gradient is multiplied by the token’s error to determine weight adjustments. Because the same network processes each time step, the backward pass revisits those shared weights serially, moving from outputs toward inputs. section 8.3

The technique becomes clearer when the recurrent computation is viewed in two ways:

• Unroll the RNN across a sentence or document to expose repeated applications of the same weight matrices. section 8.3
• Process one training example at a time or use batches; batching changes how examples are handled without changing the sequential recurrence across tokens. section 8.3

Figure 8.10 Data fed into a recurrent network

Use the recurrent-network diagram to see how token inputs and outputs are fed through successive time steps. section 8.3

How Natural Language Processing in Action, Second Edition covers it

How deep it goes

Explains chain-rule gradients, serial backward computation, unrolled shared weights, and batching, without deriving detailed equations.

Examples

Uses tokens in sentences or documents, an unrolled RNN graph, and single-sample or batched training passes.

What it assumes

Assumes familiarity with RNN forward passes and tokens while recalling the chain rule from algebra or calculus.

What you can do after

Apply it via PyTorch to adjust shared RNN weights from token-level prediction errors.

How it connects

Connects recurrent neural networks with feedforward networks, convolutional neural networks, chain-rule gradients, computational graphs, and batching.

liveBookmanning.com

Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListBackpropagation through time is a training technique that unrolls a recurrent computation across time steps and propagates gradients through the resulting sequence.

Compare how 2 books explain backpropagation through time →

Related concepts

• recurrent neural networkused by · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.